firewalld blocking DNS when running in docker-compose & sysbox runtime

README.md

Setup

docker build -t firewalld .`
docker-compose down -t 0 && docker-compose up -d
ssh 

Problem

So, it seems when this container is run with docker-compose, it can't resolve DNS using the network's DNS provider:

[root@6d5341b226fa ~]# dig +time=3 +tries=1 @10.3.0.1 google.com

; <<>> DiG 9.11.26-RedHat-9.11.26-6.el8 <<>> +time=3 +tries=1 @10.3.0.1 google.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

[root@6d5341b226fa ~]# nc 10.3.0.1 53
Ncat: Connection refused.

Now, let's see what happens if we disable firewalld:

[root@6d5341b226fa ~]# systemctl disable firewalld
Removed /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@6d5341b226fa ~]# reboot
# relogin

[root@6d5341b226fa ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enab>
   Active: inactive (dead)

[root@6d5341b226fa ~]# dig +time=3 +tries=1 @10.3.0.1 google.com +short
142.250.65.238

[root@6d5341b226fa ~]# nc 10.3.0.1 53
hi! #Sending in random data, just to show it didn't immedietely DC me.

Meanwhile, rebuilding the entire thing to get to the broken DNS state again with docker-compose down -t 0 && docker-compose up -d.

This get's interesting:

[root@99b7859d8742 ~]# dig +time=3 +tries=1 @10.3.0.1 google.com +short

; <<>> DiG 9.11.26-RedHat-9.11.26-6.el8 <<>> +time=3 +tries=1 @10.3.0.1 google.com +short
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached
[root@99b7859d8742 ~]# dig +time=3 +tries=1 @1.1.1.1 google.com +short
142.250.65.174
[root@99b7859d8742 ~]# ping 10.3.0.1
PING 10.3.0.1 (10.3.0.1) 56(84) bytes of data.
64 bytes from 10.3.0.1: icmp_seq=1 ttl=64 time=0.060 ms
64 bytes from 10.3.0.1: icmp_seq=2 ttl=64 time=0.091 ms
64 bytes from 10.3.0.1: icmp_seq=3 ttl=64 time=0.093 ms

So, it seems like somehow firewalld is just blocking 10.3.0.1 for DNS?

Now, what happens if I don't use docker-compose?

docker run --runtime sysbox-runc --ip 172.17.0.3 --rm firewalld

[root@9658d4eb62df ~]# dig +time=3 +tries=1 @192.168.3.1 google.com +short
142.250.81.238

Works fine. And yes, that's the DNS server it injects into /etc/resolv.conf

Note

Since the ssh keys are going to keep getting reset after every rebuild, you can add this to your ssh config:

Host 10.3.11.55
    User root
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null

Or, use the match processor:

Match exec "grepcidr 10.0.0.0/8 <(echo %h) &>/dev/null"
    User root
    StrictHostKeyChecking no
    UserKnownHostsFile /dev/null

docker-compose.yaml

version: '3'

services:
  firewalld:
    container_name: firewalld
    image: firewalld:latest
    runtime: sysbox-runc
    restart: unless-stopped
    networks:
      test_network:
        ipv4_address: 10.3.11.55

networks:
  test_network:
    driver: bridge
    ipam:
      config:
        - subnet: 10.3.0.0/16
          gateway: 10.3.0.1

Dockerfile

FROM centos:8
ENTRYPOINT [ "/sbin/init", "--log-level=err" ]
RUN yum install -y systemd openssh-server openssh-server bind-utils nc
RUN echo "root:1" | chpasswd
RUN echo "PermitRootLogin yes" > /etc/ssh/sshd_config
RUN yum install -y firewalld