Undesirable Docker masquerade

gistfile1.txt

Background:
===========

Running docker with --bip to support routing amongst containers across a CoreOS cluster.

Connections established outbound from containers are masqueraded to the host address. :-(

The culprit:
============

core-01 # iptables -nvL -t nat
...
Chain POSTROUTING (policy ACCEPT 271 packets, 17015 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  *      !docker0  172.18.101.0/24      0.0.0.0/0           
...
core-02 # iptables -nvL -t nat
...
Chain POSTROUTING (policy ACCEPT 539 packets, 33128 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    1    60 MASQUERADE  all  --  *      !docker0  172.18.102.0/24      0.0.0.0/0           
...

With masq rule in place:
========================

bash-4.2# tcpdump -pni eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:34:31.755617 ARP, Request who-has 172.18.101.2 tell 172.18.101.1, length 28
17:34:31.755676 ARP, Reply 172.18.101.2 is-at de:11:b5:ad:85:c9, length 28
17:34:31.755683 IP 172.17.8.102.43473 > 172.18.101.2.mdqs: Flags [S], seq 632320937, win 29200, options [mss 1460,sackOK,TS val 442448 ecr 0,nop,wscale 7], length 0
17:34:31.755709 IP 172.18.101.2.mdqs > 172.17.8.102.43473: Flags [R.], seq 0, ack 632320938, win 0, length 0

172.17.8.102 is the host address of the client container.

Remove masq rule on both nodes:
===============================

core-01 # iptables -t nat -D POSTROUTING \! -o docker0 -s 172.18.101.0/24 -d 0.0.0.0/0 -j MASQUERADE
core-02 # iptables -t nat -D POSTROUTING \! -o docker0 -s 172.18.102.0/24 -d 0.0.0.0/0 -j MASQUERADE

And now it works:
=================

bash-4.2# tcpdump -pni eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:41:51.292843 ARP, Request who-has 172.18.101.3 tell 172.18.101.1, length 28
17:41:51.292893 ARP, Reply 172.18.101.3 is-at ee:80:ff:e7:3b:32, length 28
17:41:51.292901 IP 172.18.102.4.37914 > 172.18.101.3.mdqs: Flags [S], seq 2800348254, win 29200, options [mss 1460,sackOK,TS val 879889 ecr 0,nop,wscale 7], length 0
17:41:51.292927 IP 172.18.101.3.mdqs > 172.18.102.4.37914: Flags [R.], seq 0, ack 2800348255, win 0, length 0

172.18.102.4 is the container address of the client container