The Many Potholes When Moving Webmail: Docker Networking, Reverse Proxy, iptables & DNS — A Complete Note

Posted on by Rico

Scenario

  • Host A (192.168.100.1): wwwapp (Apache reverse proxy, Docker default bridge → 172.17.0.0/16)
  • Host B (192.168.100.2): webmail (Roundcube, custom Docker mail-network → 172.24.0.0/16, published with -p 83:8000)
    Goal: Move the webmail container to Host A, and have wwwapp reverse proxy https://webmail.nuface.tw → http://192.168.100.1:83/.

Symptoms & Clues

  • Apache errors:
    • No route to host / failed to make connection to backend: 192.168.100.1:83
    • Or DNS lookup failure for: mailwebmailnuskin (service name on another network doesn’t resolve)
  • Containers lacked tools like curl/nc/ip route, making testing harder.
  • Using --network=container:wwwapp, /etc/resolv.conf showed 8.8.8.8/8.8.4.4 (legacy mode, no 127.0.0.11 embedded DNS).
  • Containers on mail-network did have /etc/resolv.conf with 127.0.0.11 (Docker embedded DNS).
  • On Host A, iptables only showed DOCKER-USER / DOCKER-FORWARD—unlike Host B’s rich set of DOCKER-ISOLATION-* and per-bridge rules.
  • Key discovery: filter INPUT did not allow Docker subnets to reach the host (default DROP). After adding rules to accept 172.17/16 and 172.24/16 in INPUT, everything worked.

Root Causes (Multiple)

  1. Container → Host traffic blocked by INPUT policy
    • Hairpin DNAT path: wwwapp container → host 192.168.100.1:83 → (nat/PREROUTING → DOCKER DNAT) → webmail container:8000.
    • The first hop is host INPUT. If INPUT default is DROP and you don’t allow traffic from docker0 / mail-network, the DNAT flow never starts.
  2. Cross-network service-name resolution misunderstanding
    • Docker’s embedded DNS resolves container names only within the same network.
    • If wwwapp is on bridge(172.17) and webmail is on mail-network(172.24), they cannot resolve each other’s container name.
    • Fix: use host IP + published port (e.g., http://192.168.100.1:83/), or connect both containers to the same Docker network.
  3. iptables backend mismatch (legacy vs nft) makes Docker rules “invisible”
    • Old host: iptables v1.4.21 (legacy)
    • New host: iptables v1.8.10 (nf_tables) (nft)
    • If Docker uses nft but you inspect with legacy (or vice versa), you’ll see “missing/incomplete” Docker rules.
  4. Bridge traffic into iptables & reply-path policies
    • If br_netfilter isn’t loaded and net.bridge.bridge-nf-call-iptables isn’t set, bridge traffic may bypass iptables.
    • In multi-bridge/PPPoE setups, rp_filter can drop replies (prefer disabling or relaxed mode).

Quick Concept Recap

  • Hairpin DNAT (loopback NAT): A container hits a host’s published port which DNATs to another container on the same host. Essentials:
    1. INPUT must permit the container’s connection to the host port (e.g., 83).
    2. FORWARD needs east-west allowances between docker0 and the target bridge (Docker usually inserts these).
    3. POSTROUTING MASQUERADE ensures proper SNAT and return path.
  • DOCKER vs DOCKER-USER:
    • DOCKER chain is managed by Docker (DNAT/SNAT and per-bridge rules).
    • Put your custom rules in DOCKER-USER. Do not flush DOCKER-* chains.
  • Container DNS:
    • Same network: embedded DNS 127.0.0.11 resolves container names.
    • Different networks: no cross-network name resolution. Use IP:PORT or connect to the same network.

Battle-Tested Diagnostic Commands

# Check if per-bridge rules exist (Docker auto-generated)
iptables -L DOCKER-FORWARD -nv
iptables -L DOCKER-ISOLATION-STAGE-1 -nv
iptables -L DOCKER-ISOLATION-STAGE-2 -nv
iptables -t nat -L DOCKER -nv | sed -n '1,120p'

# Check POSTROUTING MASQUERADE for all 172.x/16 subnets
iptables -t nat -S | grep MASQUERADE

# From inside the app container namespace
docker run --rm --network=container:<app> busybox sh -lc \
'cat /etc/resolv.conf; echo "----"; getent hosts google.com; getent hosts <backend-name>'

# From wwwapp container: test host-published port (hairpin entry)
docker exec -it wwwapp sh -lc 'wget -qO- --timeout=5 http://192.168.100.1:83/ | head -n 5' || echo FAIL

Fix Steps (Pick What You Need)

A) Minimal change: allow Container → Host in INPUT

This was the actual unlock in this incident.

Simple (what worked):

# Trust Docker default subnet
iptables -A FW-INPUT -s 172.17.0.0/16 -j ACCEPT
# Trust mail-network subnet
iptables -A FW-INPUT -s 172.24.0.0/16 -j ACCEPT

Safer (choose one style):

  • Allow by incoming interface (recommended): iptables -A FW-INPUT -i docker0 -j ACCEPT iptables -A FW-INPUT -i br-64c6b4ce2881 -j ACCEPT # mail-network bridge name
  • Or allow only necessary ports (strictest): # If host must accept 83/TCP from containers (hairpin entry) iptables -A FW-INPUT -i docker0 -p tcp --dport 83 -j ACCEPT # If host provides DNS to containers iptables -A FW-INPUT -i docker0 -p udp --dport 53 -j ACCEPT iptables -A FW-INPUT -i docker0 -p tcp --dport 53 -j ACCEPT

B) Make Docker auto-rules robust (long-term “right way”)

  1. Unify iptables backend (prefer nft): update-alternatives --set iptables /usr/sbin/iptables-nft update-alternatives --set ip6tables /usr/sbin/ip6tables-nft
  2. Send bridge traffic into iptables: modprobe br_netfilter sysctl -w net.bridge.bridge-nf-call-iptables=1 sysctl -w net.bridge.bridge-nf-call-ip6tables=1
  3. Ensure Docker manages iptables: docker info | egrep -i 'firewall|iptables' # Expect: Firewall: true
  4. Restart Docker to rebuild DOCKER-*: systemctl restart docker

C) rp_filter in multi-bridge/PPPoE

sysctl -w net.ipv4.conf.all.rp_filter=0
sysctl -w net.ipv4.conf.docker0.rp_filter=0
sysctl -w net.ipv4.conf.br-64c6b4ce2881.rp_filter=0
sysctl -w net.ipv4.conf.br.eno1.5.rp_filter=0

Reverse Proxy Tips (Two Options)

Option 1: Host IP + Published Port (no container-name DNS)

wwwapp vhost:

ProxyPass        /  http://192.168.100.1:83/
ProxyPassReverse /  http://192.168.100.1:83/
ProxyPreserveHost On
ProxyTimeout 15

With INPUT allowed and nat/DOCKER having dpt:83 -> 172.24.x.y:8000, this just works.

Option 2: Put wwwapp on mail-network

Then container-name works:

ProxyPass        /  http://mailwebmailnuface:8000/
ProxyPassReverse /  http://mailwebmailnuface:8000/

Avoids hairpin DNAT, but you must attach wwwapp to that network.

A Small, Repeatable Rule Snippet

Assumes INPUT default DROP; Docker uses nft; Docker manages the rest.

### INPUT basics
iptables -A FW-INPUT -i lo -j ACCEPT
iptables -A FW-INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

# Allow traffic from Docker bridges to host (choose one style)
# 1) By interface (recommended)
iptables -A FW-INPUT -i docker0 -j ACCEPT
iptables -A FW-INPUT -i br-64c6b4ce2881 -j ACCEPT
# 2) Or only the needed port (hairpin entry)
# iptables -A FW-INPUT -i docker0 -p tcp --dport 83 -j ACCEPT

# Other host services as needed: SSH/HTTP/HTTPS
iptables -A FW-INPUT -p tcp --dport 22  -j ACCEPT
iptables -A FW-INPUT -p tcp --dport 80  -j ACCEPT
iptables -A FW-INPUT -p tcp --dport 443 -j ACCEPT

# Drop the rest
iptables -A FW-INPUT -j REJECT --reject-with icmp-host-prohibited

Conclusion

  • Symptoms: reverse proxy fails, DNS fails, “no Internet” — looks like routing or Docker broke.
  • Real cause: filter INPUT blocked container → host traffic, so hairpin DNAT never happened; plus nft/legacy mismatches and DNS expectations.
  • One-shot fix: On Host A, allow traffic from docker0 / mail-network into INPUT (or at least open port 83/TCP).
  • Long-term: standardize on nft, enable br_netfilter, never flush DOCKER-*, put custom rules in DOCKER-USER.
  • DNS mindset: Different Docker networks can’t resolve each other’s container names. Either join the same network or use host IP + published port.

Keep this as your checklist whenever containers need to reach each other via reverse proxy on the same host and things mysteriously don’t connect.