Self-Hosting Behind CGNAT
This site is self-hosted on a server that cannot accept a single inbound connection: the ISP puts it behind CGNAT, so there is no public IP to forward ports on. The fix is a bridge—a bastion with a real public address—and a WireGuard tunnel dialed out from the homelab: clients connect to the bridge, and the bridge forwards everything back through the tunnel.
The plan §
With carrier-grade NAT, the ISP shares one public IPv4 address across
many customers: your router’s WAN address is itself
private1 1
Usually in 100.64.0.0/10, the shared address space
reserved for CGNAT by RFC 6598.
—a second NAT, outside your home,
that you don’t control. The classic recipe of port forwarding plus
dynamic DNS2 2
Shaky even without CGNAT: a rotating address is bad
for anything reputation-sensitive like mail, and every rotation means
downtime while resolvers keep serving the old IP until the TTL
expires.
dies here: forwarding only gets you through the first NAT,
and the address DDNS would publish is shared with hundreds of
strangers. Inbound connections are simply impossible.
But outbound connections still work fine—so the homelab dials out,
opening a WireGuard tunnel to the bridge and keeping it alive. The
bridge keeps only WireGuard itself and its own SSH, and forwards every
other inbound connection through the tunnel. From the outside, the
bridge is the homelab—DNS just points at it.3 3
The bridge only
pushes packets, so the smallest of servers will do. See First Steps on
a New Server for the basic setup.
Inside the tunnel, the bridge is 10.0.0.1 and the
homelab 10.0.0.2.
client admin
| |
| |
+---------------------------------------------------------+
| public Internet |
+---------------------------------------------------------+
| | |
| | |
+---------------------+ +--------------+ |
| bridge 10.0.0.1 | | Cloudflare | | homelab's own
| DNAT * -> homelab | +--------------+ | outbound traffic
+---------------------+ ^^ |
^^ || |
|| WireGuard || cloudflared |
|| (all ports) || (backup SSH) |
vv vv |
+---------------------------------------------------------+
| homelab 10.0.0.2 (behind CGNAT) |
+---------------------------------------------------------+
| ^
| self-check via bridge---or reboot |
+-------------------------------------+
Both tunnels are dialed out from the homelab—only outbound works behind CGNAT. The admin SSHes in through the bridge like any other client (port 22 is forwarded with the rest); the homelab’s own outbound traffic never crosses it, only replies to forwarded connections do. The Cloudflare backup tunnel and the watchdog loop are covered at the end.
The tunnel §
On both machines, install WireGuard and generate a keypair; exchange the public keys—the private ones never leave their machine.
The bridge’s entire setup lives in /etc/wireguard/wg0.conf
[Interface]
Address = 10.0.0.1/24
PrivateKey = <bridge-private-key>
ListenPort = 51820
PostUp = ...
PostDown = ...
[Peer]
PublicKey = <homelab-public-key>
AllowedIPs = 10.0.0.2/32
where PostUp enables IP forwarding and installs the five iptables
rules that do the forwarding, tying their lifetime to the
tunnel’s4 4
PostDown mirrors PostUp, undoing every command.
ens3 is the bridge’s public interface—find yours with ip a and
adjust.
sysctl -w net.ipv4.ip_forward=1 net.ipv4.conf.ens3.route_localnet=1
iptables -t nat -A PREROUTING -i ens3 -p udp --dport 51820 -j RETURN
iptables -t nat -A PREROUTING -i ens3 -p tcp --dport 2222 -j RETURN
iptables -t nat -A PREROUTING -i ens3 -j DNAT --to-destination 10.0.0.2
iptables -A FORWARD -i wg0 -o ens3 -s 10.0.0.2 -j ACCEPT
iptables -A FORWARD -i ens3 -o wg0 -d 10.0.0.2 -j ACCEPT
Two RETURN rules keep WireGuard (51820/udp) and the bridge’s own
SSH (2222/tcp) local; the catch-all DNAT rewrites everything
else—port 22 included—to the homelab’s tunnel address; and two
FORWARD accepts let that traffic flow both ways.5 5
Note there is no
MASQUERADE: conntrack reverses the DNAT on the way out, and the
homelab routes its replies back through the tunnel. Forwarded services
see the real client IP—something proxies and third-party tunnels
can’t offer. This holds for services that terminate on the homelab
itself; a service behind a second NAT hop—anything in a Docker
container, say—needs one more touch on the homelab, noted below.
Everything happens inside the kernel: netfilter does
the rewriting, and no userspace process ever touches a packet.
The bridge’s own sshd listens on 2222 precisely so that port 22 can be
forwarded with everything else: ssh alvarezrosa.com lands on the
homelab, ssh -p 2222 on the bridge.6 6
Add Port 2222 to the
bridge’s /etc/ssh/sshd_config before bringing the tunnel up—the
moment the DNAT rule takes effect, port 22 belongs to the homelab.
The homelab side dials out and answers. Its /etc/wireguard/wg0.conf
[Interface]
Address = 10.0.0.2/24
PrivateKey = <homelab-private-key>
Table = off
PostUp = ...
PostDown = ...
[Peer]
PublicKey = <bridge-public-key>
Endpoint = 213.32.19.229:51820
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 25
where PostUp sets up policy routing.
ip route add default dev wg0 table 200
ip rule add from 10.0.0.2 table 200
iptables -t mangle -A PREROUTING -i wg0 -m conntrack --ctstate NEW -j CONNMARK --set-mark 0x1
iptables -t mangle -A PREROUTING -m conntrack --ctdir REPLY -j CONNMARK --restore-mark
ip rule add fwmark 0x1 lookup 200 pref 1000
Each line earns its place: AllowedIPs = 0.0.0.0/0 accepts forwarded
clients from anywhere on the Internet; Table = off stops wg-quick
from hijacking all of the homelab’s traffic through the
bridge7 7
Without it, wg-quick would install a default route
matching AllowedIPs.
; the policy routing sends only replies of
forwarded connections—packets from 10.0.0.2—back through the
tunnel; and PersistentKeepalive keeps the CGNAT’s idle UDP mapping
alive, so the bridge can always reach in.
This source-based rule is enough for services that listen on the
homelab itself, but it quietly breaks the moment a forwarded service
lives behind a second NAT—a Docker container, most commonly. Such a
service sits on its own address (say 172.19.0.4), reached by a
further DNAT; and the reply’s source is rewritten back to 10.0.0.2
only in POSTROUTING, after the routing decision is made. At
routing time the packet still reads from 172.19.0.4, misses the
rule, falls through to the main table, and leaks out the WAN
interface—straight into the CGNAT, where it dies. The cure is to
route by the connection rather than by an address that isn’t settled
yet—the last three lines above. The first marks every new
connection arriving on wg0; the second restores that mark onto the
packets travelling the other way, in PREROUTING, before the
routing decision; and the fwmark rule sends whatever carries the
mark through the tunnel. Routing by mark instead of by source keeps
the real client IP intact—a MASQUERADE on the homelab would have
been shorter, but it would rewrite that address away.
Enable the tunnel on both machines with
sudo systemctl enable --now wg-quick@wg0 and verify the handshake
with sudo wg. Then the real test: from outside, any connection to
the bridge’s public IP should land on the homelab. Point your DNS
records at the bridge and the homelab is, for all practical purposes,
on the public Internet.
The detour adds latency: this bridge sits in France, the homelab in northern Spain, and the tunnel adds ~37 ms of RTT to every connection.8 8 Amusingly, pinging the bridge’s public IP from the homelab reports ~74 ms—exactly double. The echo request is DNAT’d back through the tunnel to the homelab itself, so every packet crosses the tunnel twice. Not a problem in practice: with heavy optimization and a CDN absorbing most requests, this site—served through this very tunnel—is among the fastest on the web.
Plan for failure §
Two single points of failure, and a plan for each.
If the bridge dies, the tunnel dies with it—so keep a way into the
homelab that bypasses it entirely. I run a Cloudflare Tunnel:
cloudflared uses the same dial-out trick, outbound-only on both ends,
so it also works behind CGNAT.9 9
Tailscale fills the same role.
It
exposes the homelab’s sshd at a hostname of its own, and a
ProxyCommand in the client’s \~/.ssh/config connects through
it—whatever happens to the bridge, ssh homelab2 still gets in.
If the homelab dies, no tunnel will save you—the machine to reboot
is the one you can’t reach. So it watches itself with a root cron
job—0 5 * * * ssh ssh.alvarezrosa.com || reboot—that SSHes to
its own public hostname, out through CGNAT to the bridge and back in
through the tunnel, the whole chain end to end, and reboots if that
fails.10 10
A bridge outage also trips this check and reboots a
perfectly healthy homelab—an acceptable false positive, since a
reboot is harmless.
That’s the whole trick: one cheap bridge, one tunnel, five iptables rules—and a server behind CGNAT serves the public Internet, this very page included.
Subscribe§
My mailing list is free, occasional, and covers a variety of topics. I will never sell or share your email address.
Have feedback? Email me at david@alvarezrosa.com.