Pete's Log: Residential Gateway Bypass

Entry #2545, (Coding, Hacking, & CS stuff)
(posted when I was 46 years old.)

To anyone who just wants to know: I can confirm that as of August 2024, the opnatt.sh script at MonkWho/pfatt works in bridge mode with OPNsense 24.7_9 and a BGW210.

The repo above hasn't been updated in four years and I had hoped to find recent confirmation myself before I proceeded, but couldn't find anything definitive. Anyway, for more details on my journey, read on.

Before today, the core of our home network looked something like this:

Network diagram, showing the internet on the left, connected in series to two blue boxes, first one labeled ONT and second one labeled RGW. The rest of the diagram is red. Connected to the RGW is a box labeled firewall/router which in turn is connected to a box labeled Laundry Room Switch. That box is connected to three other boxes: Garage Switch, AP, Office Switch

Where the blue boxes belong to AT&T and the red boxes belong to us. ONT is the Optical Network Terminal, where the fiber into our house terminates. RGW is the Residential Gateway. The RGW and ONT do some authentication handshakes before the internet connection can come up, which is why our firewall was plugged into the RGW instead of the ONT. The RGW does have a "passthrough" mode in which it can hand off its assigned IP to another router, which is what I was using.

The Firewall / Router is a pfSense-based Netgate device. I've been wanting to replace it for some time now. This weekend I finally did. There were two reasons:

  • It was slightly underpowered for our 1Gbps internet connection
  • While our Netgate device has three ports on it, those three ports actually connect to a little mini-switch within the device, and thus the CPU doesn't see physical interfaces for those three ports. For the opnatt.sh bypass to work, I need three ports that the CPU can actually see as physical interfaces. And I really wanted to bypass the RGW, for a handful of reasons:
    • The BGW210 has a relatively small NAT state table that still limits NAT even in passthrough mode (though this never actually impacted us)
    • The BGW210 is assigned an IPv6 /60 delegated prefix, but only hands off a /64 prefix to the router, meaning I can only assign IPv6 to one of my subnets. I really want to learn IPv6 better so I really want to get my hands on the full /60.
    • It's an extra hop all our network packets have to go through and it would just be nice if they didn't have to.

So after pondering it for a couple years now, I finally treated myself to a Protectli Vault VP2420 for my birthday. In addition to being plenty powerful, it has four physical interfaces and also the option to add a 4G LTE modem at some point for backup connectivity.

So now the network looks like this:

Network diagram, showing the internet on the left, connected to a blue box labeled ONT. Connected to the ONT is a red box labeled firewall/router. Connected to it are two boxes. A blue box labeled RGW and a red box labeled Laundry Room Switch. That box is connected to three other boxes: Garage Switch, AP, Office Switch

The opnatt.sh script filters the authentication packets and passes them between the ONT and RGW, but otherwise no traffic passes through the RGW. The new router gets assigned the full /60 prefix so I can give all my subnets IPv6. Well, technically I can give up to 16 subnets IPv6, but I currently only have six subnets, so there's still room to grow. I am very pleased.

A bullet list of a few more notes:

  • Speed tests (over ethernet) now give me well over 900 Mbps up and down. Before my updates they tended to max out in the high 700 Mbps range. And now during speed tests the firewall CPU utilization is just under 20% while the Netgate would peg to 100%.
  • The documentation and UI for acme-client/LetsEncrypt in OPNsense is baffling in how bad it is.
  • I did make one mistake while configuring opnatt.sh. Instead of using the MAC address on the RGW port that the ONT was plugged into, I used the MAC address of the RGW port that the router was plugged into. So using tcpdump I could see that the EAP/802.1x authentication was properly happening between the RGW and ONT, but my router would not get any response to its DHCP requests until I fixed the MAC address.
  • This was a big "little victory" and I've been feeling very happy since getting this working.

Here's the Vault and ONT, connected directly to each other:

Firewall and ONT