setup shorewall for docker networking beyond the default bridge network, e.g., for docker-compose

00-docker-shorewall.md

Docker(-compose) with shorewall

The shorewall documentation explains in http://shorewall.org/Docker.html how to configure shorewall for use with docker. The problem with the configuration is that it only allows connections from the host to the main bridge docker0. Connections to other networks on dynamically created bridges, with names starting by default with br-, is blocked. Instead of the recommended contents of /etc/shorewall/interfaces, use wild-card interface names as follows:

#ZONE	INTERFACE	OPTIONS
#dock	docker0		bridge     # disabled default recommendation
dock 	docker0		physical=docker+,routeback=1
dock 	br		physical=br-+,routeback=1

This declares interfaces with names starting with docker, including the default docker0, and starting with br- to be in the dock zone.

For the rest of the configuration, follow the shorewall documentation as is.

This setup fixes problems running composite apps set up manually or using docker-compose.

interfaces

###############################################################################
?FORMAT 2
###############################################################################
#ZONE   INTERFACE       OPTIONS
net     eth0            dhcp,tcpflags,nosmurfs,routefilter,logmartians,sourceroute=0
#dock   docker0         bridge
dock    docker0         physical=docker+,routeback=1
dock    br              physical=br-+,routeback=1
# and others ...

policy

###############################################################################
#SOURCE         DEST            POLICY          LOG LEVEL       LIMIT:BURST

$FW             net             ACCEPT

dock            $FW             REJECT
dock            all             ACCEPT
# Some applications or use case require uncommenting the next line
$FW             dock            ACCEPT

# THE FOLLOWING POLICY MUST BE LAST
all             all             REJECT          info

shorewall.conf

# apply the change below, as

# ...

# LN:
#DOCKER=No
DOCKER=Yes

# ...

zones

###############################################################################
#ZONE   TYPE    OPTIONS                 IN                      OUT
#                                       OPTIONS                 OPTIONS
fw      firewall
dock    ipv4
# and others ...

These are the other relevant parts of my configuration, maybe I miss something?

root@dk1:~# cat /etc/shorewall/interfaces
net ens3 - routefilter,tcpflags
vpn tun0
dock docker0 - physical=docker+,routeback=1
dock br - physical=br-+,routeback=1

root@dk1:~# cat /etc/shorewall/interfaces
net ens3 - routefilter,tcpflags
vpn tun0
dock docker0 - physical=docker+,routeback=1
dock br - physical=br-+,routeback=1

root@dk1:~# cat /etc/shorewall/policy
net fws DROP
fws all ACCEPT
vpn fws ACCEPT
dock fws REJECT
dock all ACCEPT
all all DROP

I don't know if this helps you @MichaelUray, but I experienced that the order of starting Shorewall and Docker is important (also when restarting one of them). If I remember right, start Shorewall first, then Docker. Try to verify this.

You are right, it helps to restart docker after shorewall was restarted, but I actually don't want to restart Docker after every change on the firewall settings.

I don't know of a way to avoid restarting docker after restarting shorewall. I made it a habit to restart both together - but I agree it is a minor pain. There seems to be a way to tell systemd to restart docker automatically after restarting shorewall, got to look into the configuration changes required. This will still restart all running containers, though.

This will still restart all running containers, though.

Thank you, this is what I actually want to prevent.

If configured with the DYNAMIC_BLACKLIST=Yes option, you can use shorewall open, shorewall close and other commands for non-persistent changes (see https://shorewall.org/manpages/shorewall.html). You still have to update the config files to be sure changes are persistent. But the persistent rules won't get verified until you restart. I recommend to shorewall compile, which catches syntax errors but won't change the active rules.

In practice, I find firewall rules pretty stable and the restart issues don't arise often. Using the shorewall commands is OK for testing which ports are needed when setting up new apps. I would still do one restart with the persistent configuration at the end. Bugs that won't show up until the next reboot are worse than one container restart.

Thank you, it works.

(I think my issue is related to the ones mentioned above, so I'll add my issue here)

I'm using Docker 19.03.6 and shorewall 5.1.12.2 and encountered situations where (creating and) starting a docker container caused the following error:

Creating network "acc_default" with the default driver
ERROR: unable to insert jump to DOCKER-ISOLATION-STAGE-1 rule in FORWARD chain: (iptables failed: iptables --wait -I FORWARD -j DOCKER-ISOLATION-STAGE-1: iptables v1.6.1: Couldn't load target `DOCKER-ISOLATION-STAGE-1':No such file or directory

How to reproduce:
$ sudo service shorewall stop
$ suso service docker stop
$ sudo docker start
$ sudo shorewall start
<< when checking iptables -L, chain DOCKER is present, chains DOCKER-ISOLATION-STAGE-1 and DOCKER-ISOLATION-STAGE-2 are not! >>
$ docker-compose up -d
<< now the error as mentioned above occurs >>

I noticed that the chains DOCKER-ISOLATION-STAGE-1 and DOCKER-ISOLATION-STAGE-2 (including references in other chains) were removed after the command "shorewall restart". As they didn't contain any rules and everything kept working (also after stopping starting containers), I didn't see this as an issue. However, when starting a new container (e.g. after it has been removed) it will fail with the error mentioned above.
When executing shorewall with the trace option, I can see mentioned DOCKER-INGRESS and DOCKER-ISOLATION, but never DOCKER-ISOLATION-STAGE-1 or DOCKER-ISOLATION-STAGE-2.

With some further test I discovered that without the DOCKER-ISOLATION-STATE-x chains, I am able to start any docker container, but can NOT create a docker network!

So my questions are:

1. Has docker changed their iptables (are the ...-STAGE-.. chains new) and is shorewall not ready for them yet?
2. If so, when is a new version of shorewall to be expected?
3. Most important: what do I need to do to allow shorewall to modify iptables without having to restart the docker daemon? (we are using mrlesmithjr/ansible-shorewall to configure shorewall via ansible)

In my experience, shorewall has to be started before other tools that manipulate the iptables. On Debian, I can verify that the dependencies in the systemd unit files are set up in a way that makes sure that shorewall starts before docker. If you change the order of startup to

$ sudo shorewall start
$ sudo docker start

things should work as expected.

BTW: If you have other services manipulating iptables, e.g., fail2ban, make sure they get restarted after shorewall gets restarted.

Thanks for your feedback, @lukasnellen.
I was under the impression that shorewall would preserve docker related chains and rules when being restarted, but apparently it doesn't :-(
That means a firewall modification will still require a restart of the docker daemon (which means an outage for our production services).

I was able to re-create the Docker Isolation chains, but the Docker chain itself is a bit more tricky...

In practice, the problem is that restarting docker restarts all running containers. This makes frequent changes (or trying changes) on a production machine difficult. This is not a problem for my setup here, where changes are limited (and I have a non-production machine for trying out things). This meant I never had to look into the use of shorewall open and shorewall close as means of run-time modification of shorewall's configuration (more in a comment higher up).

Thank you very much

Thanks for your helpful hints.
I am using docker-compose and shorewall. When setting rules in /etc/shorewall/rules they are not triggered.
To be able to restrict access to the docker machines I need to insert rules into the "DOCKER-USER" chain. Any idea how I can manage that with shorewall?

Instead of using the DOCKER-USER chain, I set up rules in shorewall that filter connections to or from the dock zone. For more fine-grained control, you can use some of docker's advanced options:

• create separate networks, maybe with a named bridge so you can create a special zone on that bridge
• set the IP of the containers in docker run or in docker-compose.yml
• user shorewall's dynamic zones (https://shorewall.org/Dynamic.html), put your container's IP into the correct, dynamic zone, and filter on those zones.

The point here is to control the interface or IP address used by a container. That way, you can write specific filter rules in shorewall w/o having to inspect the container's configuration, which might change when the container gets re-created. This is very similar to how I set up filtering for full-blown virtual machines.

The alternative is to use iptables to manipulate the DOCKER-USER table, which is not managed directly by shorewall.

related to above comments about iptables rules on debian, it seems that the versions of shorewall and docker in debian stable and above do not work (at least not without always restarting docker after shorewall is restarted)

i rebuilt the debian shorewall packages (shorewall + shorewall-core) with the 5.2.4 version and this fixed the problem

i have raised an issue on the shorewall gitlab issue tracker

https://gitlab.com/shorewall/debian/-/issues/1

Thank you very much.

Great post!
However in my case it's missing a vital detail !
I disabled docker iptables in /etc/docker/daemon.json
{
"iptables":false
}
And after that all works Great! Docker containers are accessible from net if enabled in shorewall ( rules, zones, policy ) etc !!
Regards,
Harry

This line is giving a Broadcast error when starting Shorewall 5.2.3.2

dock docker0 physical=docker+,routeback=1

ERROR: Invalid BROADCAST address /etc/shorewall/interfaces

I added different and it worked as expected.

dock docker physical=docker+,routeback=1

The actual name should not matter, since the wildcard physical=docker+ is there to match any interface with a name starting with docker.

Details might also depend on the way how the distribution sets up networking and on the order in which components get started.

Disclaimer: My production machines are debian, versions 9-11. So details might vary. I'm glad to get additional information on what to change/fix on other distributions.

I am using debian 10 and just cannot get this to work -- the only way to get it working for me, and I still don't think its right, is using::

dock    docker0
dock    br

shorewall version
5.2.3.2

Did you try running shorewall compile? This gives better diagnostics than trying to restart the service. And it allows you to detect and fix problems rather than leaving the firewall in an unreliable or diabled state (especially when restarting).

I recently migrated my debian 10 docker host to 11 and don't have a debian 10 docker host at hand to verify the behaviour.

shorewall compile

Compiling using Shorewall 5.2.3.2...
Processing /etc/shorewall/params ...
Processing /etc/shorewall/shorewall.conf...
Loading Modules...
Compiling /etc/shorewall/zones...
Compiling /etc/shorewall/interfaces...
ERROR: Invalid BROADCAST address /etc/shorewall/interfaces (line 191)

Do you have the format specifier at the top of the interfaces file? The line

?FORMAT 2

is required. If I leave it out, I can reproduce the error message about the BROADCAST address in my setup. It almost looks like a comment i the file I have included at the top.

Crap... missed that. It's working now!! Thanks mate.

Great post! However in my case it's missing a vital detail ! I disabled docker iptables in /etc/docker/daemon.json { "iptables":false } And after that all works Great! Docker containers are accessible from net if enabled in shorewall ( rules, zones, policy ) etc !! Regards, Harry

@grharry This is the ""detail"" I was missing! Thank you so much!

Thanks! New docker user and this gist and comments allowed me to connect to running containers.

But now running containers cannot reach the wider internet. How do I do that?

Edit: resolved with help from serverfault.com/a/631877 specifically creating a /etc/shorewall/masq file and turning on ip forwarding.

Has anyone succeeded in restricting network access to containers for the entire Internet and only allowing access to a specific list of addresses (ipset)?

You saved my day, thank you so much!

Hero!