Permissive forwarding rule leads to unintentional exposure of containers to external hosts

From self[at]sungpae.com Mon Nov  8 16:59:48 2021
Date: Mon, 8 Nov 2021 16:59:48 -0600
From: Sung Pae <self[at]sungpae.com>
To: [email protected]
Subject: Permissive forwarding rule leads to unintentional exposure of
 containers to external hosts
Message-ID: <YYmr4l1isfH9VQCn@SHANGRILA>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="QR1yLfEBO/zgxYVA"
Content-Disposition: inline
X-PGP-Key: fp="4BC7 2AA6 B1AE 2B5A C7F7  ADCF 9D1A A266 D2BC 9C2D"
X-TUID: Avm8Mn+0Qq5s


--QR1yLfEBO/zgxYVA
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Hello,

The documentation for "docker run --publish" states:

> Note that ports which are not bound to the host (i.e., -p 80:80 instead of
> -p 127.0.0.1:80:80) will be accessible from the outside. This also applies
> if you configured UFW to block this specific port, as Docker manages his own
> iptables rules.

https://docs.docker.com/engine/reference/commandline/run/#publish-or-expose-port--p---expose

The statement above is accurate, but terribly misleading, since traffic
to the container's published ports from external hosts will still be
forwarded due to an explicit forwarding rule added to the DOCKER chain:

    # iptables -nvL DOCKER
    Chain DOCKER (2 references)
     pkts bytes target prot opt in       out     source    destination
        0 0     ACCEPT tcp  --  !docker0 docker0 0.0.0.0/0 172.17.0.2  tcp dpt:80

An attacker that sends traffic to 172.17.0.2:80 *through* the docker
host will match the rule above and successfully connect to the
container, obviating any security benefit of binding the published port
on the host to 127.0.0.1.

What's worse, users who bind their published ports to 127.0.0.1 operate
under a false sense of security and may not bother taking further
precautions against unintentional exposure.

## Proof of Concept

Here is a simple proof of concept:

1. [VICTIM] Start a postgres container and publish its main port to
   127.0.0.1 on the host.

   [email protected]# docker run -e POSTGRES_PASSWORD=password -p 127.0.0.1:5432:5432 postgres

2. [ATTACKER] Route all packets destined for 172.16.0.0/12 through the
   victim's machine.

   [email protected]# ip route add 172.16.0.0/12 via 192.168.0.100

3. [ATTACKER] Discover open ports on the victim's internal docker networks.

   [email protected]# nmap -p5432 -Pn --open 172.16.0.0/12
   Starting Nmap 7.92 ( https://nmap.org ) at 2021-11-05 15:00 CDT
   Nmap scan report for 172.17.0.2
   Host is up (0.00047s latency).

   PORT     STATE SERVICE
   5432/tcp open  postgresql

4. [ATTACKER] Connect to the victim's container.

   [email protected]# psql -h 172.17.0.2 -U postgres
   Password for user postgres:

## Scope of Exposure

Port publishing in docker and docker-compose is a popular way to expose
applications and databases to developers in a cross-platform development
environment.

Web searches for the pitfalls of "--publish", as well as discussions
with other developers, suggest that Docker users who are aware of the
security implications of port publishing also believe that specifying an
IP address to bind on the host will effectively constrain access to the
service they are attempting to share. This is a reasonable conclusion
that can be drawn from the documentation, but the reality is that simply
publishing a port exposes a container to external machines regardless of
the IP address bound on the host.

Github contains tens of thousands of projects that publish container
ports to "127.0.0.1:xxx:xxx":

* https://github.com/search?q=docker+run+%22-p+127.0.0.1%3A%22&type=code
* https://github.com/search?q=docker+run+%22--publish+127.0.0.1%3A%22&type=code
* https://github.com/search?p=5&q=%22127.0.0.1%3A5432%3A5432%22&type=Code
* https://github.com/search?q=%22127.0.0.1%3A15432%3A5432%22&type=code
* https://github.com/search?q=%22127.0.0.1%3A3306%3A3306%22&type=Code
* https://github.com/search?p=5&q=%22127.0.0.1%3A8080%3A80%22&type=Code
* And many more!

Here is a sampling of commit messages that specifically mention the
security rationale behind publishing to "127.0.0.1":

https://github.com/rubyforgood/abalone/commit/764a619babc7ac05fe9fe6edc63e9128a2c86af3

> Forward the "db" service's port to the host's loopback interface, so
> that a developer could choose to use docker-compose only for a container
> to run the database while running all the Ruby processes on their host
> computer. "127.0.0.1:5432:5432" was chosen over "5432:5432" so that the
> PostgreSQL would not be available to all other computers on the host
> computer's network (say, a coffee shop wifi).

https://github.com/MayankTahil/pref/commit/f3056408867a227e9ff6b338c51ef37d605f5dad

> [SECURITY] Limit port export to localhost
>
> It's prevents leak private developed projects vie Eth & Wi-Fi interfaces.
> You now must use `localhost` host or use host mapped directly to 127.0.0.1

https://github.com/open-edge-insights/eii-core/commit/7a85ab8ed818af73a83489554eb5737394a4cf0c

> Docker Security: Port mapping and default security options
>
> Changes:
>
> 	1) Provide secuiry options in docker-compose file related to selinux and resticted privilages
> 	2) Set HOST_IP as Environment Variable in Compose startup
> 	2) Bind all ports to either 127.0.0.1 or Host IP

## Mitigation

While the unintentional exposure of published container ports can be
mitigated by constraining access to containers in the DOCKER-USER chain,
my observation is that most Linux users do not know how to configure
their firewalls and have not added any rules to DOCKER-USER. The few
users that do know how to configure their firewalls are likely to be
unpleasantly surprised that their existing FORWARD rules have been
preceded by Docker's own forwarding setup.

In light of this, an effective mitigation should:

1. Restrict the source addresses and/or interfaces that are allowed to
   communicate with the published container port.

   For example, "docker run -p 127.0.0.1:5432:5432" creates the
   following rule in the DOCKER chain:

   Chain DOCKER (2 references)
    pkts bytes target prot opt in       out     source    destination
       0 0     ACCEPT tcp  --  !docker0 docker0 0.0.0.0/0 172.17.0.2  tcp dpt:5432

   It should, however, restrict the source ip address range to
   127.0.0.1/8 and the in-interface to the loopback interface:

   Chain DOCKER (2 references)
    pkts bytes target prot opt in out     source      destination
       0 0     ACCEPT tcp  --  lo docker0 127.0.0.1/8 172.17.0.2 tcp dpt:5432

   The values of "127.0.0.1/8" and "lo" can be retrieved from the
   interface on which 127.0.0.1 is defined. For instance, if a machine
   has an IP address of 192.168.0.100 on a /24 network on eth0 and the
   user runs "docker run -p 192.168.0.100:5432:5432", we would expect to
   see the following:

   Chain DOCKER (2 references)
    pkts bytes target prot opt in   out     source         destination
       0 0     ACCEPT tcp  --  eth0 docker0 192.168.0.0/24 172.17.0.2 tcp dpt:5432

2. Default to "127.0.0.1" when a bind address is not supplied to "--publish".

   This is a breaking change, but it should have been the default from
   the beginning.

## Conclusion

Docker port publishing is an *extremely* popular feature, and at
present, virtually all users that use containers with published ports
are exposed to attackers that have noticed the oversight outlined in
this email.

I have not noticed any discussion online of attackers using custom
routes to gain access to containers, but it is an obvious attack, and
perhaps unfortunately, I posted a comment about this vulnerability in a
related Github issue:

https://github.com/moby/moby/issues/22054#issuecomment-962202433

Thank you for your attention to this.

Sung Pae
https://github.com/guns

--QR1yLfEBO/zgxYVA
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEES8cqprGuK1rH963PnRqiZtK8nC0FAmGJq+IACgkQnRqiZtK8
nC3TvBAAka0sVXX4X2k8BIzVUoojrM1OkOBzAZl76cdI1Zmv4P6sp/zmkR7iE5eV
lUQ57cLwnalbbn9e0QyVA2/jcuB96cx8bKL8jy+JnJ0IuQ4VUYEWGTkLORIojDRJ
I8imGY83Bz4fyffoMUxG3DBeuJJCOHIHFbcoijI4xYPz2ujY3KR0vC0UYxcZLv92
bD1thh/bFXaPOPBHlVCUB9hFq1/JZ27XaH9GZ7X7TeuOp25JriU1h3U/A6gsGTkK
OBOjRVJV30tDnsVZa8TBvL27JfLGyRvGACpnOhaozSvVgePERDBeMsH6bjDNzWEs
mb9QIsxA6brZdJdH1uXJDM36nhG1eT3OM7jrZzI76+7FT2yzrQcKsk2Oes0t9ZCq
wyZbVoZGExam2bPiWvu9XJVb9TPwKpxXpLPyiuFZrlrOaBfDV1ZqFMSBXZJxFOuu
PEysiYTUpq+FufGJxH5JqWERLh79TV/f+654DG/UtOas+A7Rjy6hF9OsDXDWzpz/
lo7w3OKaXqvNZ2ysL8ihHp963fFLPkhMn2JAOBsoFa3s/hCCBwFJjxHnzF9gNRhZ
cr9f3wlk6IVJMSARPJsZCD+g5uaU1gzDbndem3SlMLjkJ4D6rLoZ3zhmkddjKhsv
CLBL6R7nEmuBcb4e97EVmCnYR8221uXmqvc2bQwPpeTLeGMG5BQ=
=+qVS
-----END PGP SIGNATURE-----

--QR1yLfEBO/zgxYVA--

The problem is that approximately dozens of users worldwide know that this is necessary when using --publish 127.0.0.1:port:port. A default behavior that violates assumptions about binding to 127.0.0.1 and undermines secure firewalls is an issue that requires fixing even if can be mitigated by the user.

I agree. I consider myself to be an advanced user and even I didn't think docker would expose services that I'd assume were bound to the lo interface, but are actually accessible remotely by default. And it's insidious because if you just portscan your machine you wouldn't see it either.

I might try to open a PR for their documentation page on iptables to clarify the behavior and a stopgap fix

-A DOCKER-USER -o br-+    -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment DOCKER-inbound  -j RETURN
 -A DOCKER-USER -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment DOCKER-inbound  -j RETURN
-A DOCKER-USER -i br-+                                               -m comment --comment DOCKER-outbound -j RETURN
-A DOCKER-USER -i docker0                                            -m comment --comment DOCKER-outbound -j RETURN
-A DOCKER-USER                                                                                            -j REJECT

Is that enough? Could an attacker on the local network mess up with conntrack (sending forged SYN, SYN+ACK, ACK)? Is it not necessary to add something like:

iptables -t raw -A PREROUTING -m rpfilter --invert -j DROP