Overlay network and static IPs for Docker containers

overlay-network-and-static-ips-for-docker-containers.md

I don't expect moby/moby#24170 to be solved by Docker team on near future so this gist is about looking for least ugly workarounds which are available today and on near future.

What is possible on Docker 19.03

Network

It is possible to create overlay network with use user specified subnet. On this example I create network which is size is two C -class networks ( IP range 10.0.0.0 - 10.0.1.255 ) and force Docker to use IPs from second part of it ( 10.0.1.0 - 10.0.1.255 ).

That way I can make sure that IPs 10.0.0.2 - 10.0.0.254 can be specified to containers and they do not collide with IPs which Docker engine assign for service(s)/container(s).

docker network create --driver overlay --subnet 10.0.0.0/23 --ip-range 10.0.1.0/24 --gateway 10.0.0.1 --attachable test

Containers

By default containers IP addresses are controlled by Docker engine and those cannot be changed inside of container. Luckily that behavior can be overridden by adding NET_ADMIN capability for container.

Custom Docker image

So first we need create custom docker image which is able set IP for container when it starts. That we can do example with Dockerfile like this:

FROM ubuntu
RUN apt-get update \
        && apt-get install -y net-tools iputils-ping
COPY /start.sh /
ENTRYPOINT /start.sh

and start script like this:

#!/bin/bash
if [[ -f "${STATIC_IP}" ]]; then
        echo "Using default IP from Docker"
else
        echo "Found static IP: ${STATIC_IP} using it"
        ifconfig eth0 ${STATIC_IP} netmask 255.255.255.0 up
fi
sleep infinity

Then just build that with command:

docker build . -t static-ip

Starting container

Then we can start two containers with commands like these which connects then to overlay network and changes IP addresses inside of the to static ones.

docker run --name test1 --detach=true --rm --cap-add=NET_ADMIN --network=test -e STATIC_IP=10.0.0.11 static-ip
docker run --name test2 --detach=true --rm --cap-add=NET_ADMIN --network=test -e STATIC_IP=10.0.0.12 static-ip

What will be possible on next version

Above solution solves two issues:

• It sets static IPs for containers.
• It allow containers communicate between Docker nodes using overlay network.

However those containers are not swarm services which means that you lose some control to them. Luckily I have already implemented docker/cli#1940 and I'm expecting that it will get merged and released as part of next Docker version.

That makes possible to use swarm stack like this:

version: "3.7"
services:
  test1:
    image: static-ip
    environment:
      STATIC_IP: 10.0.0.11
    networks:
      - test
    cap_add:
      - NET_ADMIN
  test2:
    image: static-ip
    environment:
      STATIC_IP: 10.0.0.12
    networks:
      - test
    cap_add:
      - NET_ADMIN

networks:
  test:
   external: true
   name: test

and deploy it with command:

docker stack deploy -c static-ip.yml static-ip

@nrodday honestly I wrote this so many years ago that I don't remember if actually tested multi node setup. We eventually found solution make applications working with dynamic IPs so there was not need for that.

However if there would have mandatory use case for this then I probably would be testing multi network or multi IP per network config depending on what is reason for static IPs.

Those who are familiar with Golang can also build ipam plugin for this purpose. There is somekind of example available on https://github.com/ishantt/docker-ipam-plugin

I found a workaround and I am adding it here for reference:

docker network disconnect OVERLAY_NETWORK_NAME CONTAINER_NAME
docker network connect --ip 172.30.0.101 OVERLAY_NETWORK_NAME CONTAINER_NAME

This also updates the docker overlay network configuration such that inter-node communication is possible. This command has to be executed from within every node for the respective containers running on that node. One could build some scripts around to automate the reassignment.

From my point of view, "docker stack deploy --compose-file FILE_NAME STACK_NAME" should respect the ipv4_address attribute within the compose file for the first replica and go for additional replicas with dynamic IPs, which it currently does not.
networks:
bgp_net:
ipv4_address: 172.30.0.101

Anyway, via the workaround one is able to get a static IP into a swarm via an overlay network.