Docker Network Overlap: Root Cause Analysis

docker-network-overlap-investigation.md

Docker Network Overlap: Root Cause Analysis

Error

failed to start daemon: Error initializing network controller: error creating default "bridge" network:
cannot create network 0866bc6b5828... (docker0): conflicts with network a20d7f349c72...
(br-a20d7f349c72): networks have overlapping IPv4

Investigation

Inspected the cached Docker network state database at /var/lib/docker/network/files/local-kv.db (BoltDB) from a failing task container. It contains a single libnetwork bucket with 4 stored networks:

Name	Type	Subnet	Bridge Interface	Created
bridge	bridge	172.17.0.0/24	docker0	2026-03-20T06:19:04Z
mint-workspace_default	bridge	172.17.1.0/24	br-a20d7f349c72	2026-03-20T06:19:05Z
host	host	—	—	2026-02-26T04:07:57Z
none	null	—	—	2026-02-26T04:07:57Z

The mint-workspace_default network was created by Docker Compose (com.docker.compose.project: mint-workspace, Compose version 2.34.0). Both bridge networks were allocated /24 subnets from the default-address-pools base of 172.17.0.0/16.

Root Cause

1. A task runs with Docker enabled. The Go agent bind-mounts a daemon.json with:

   {
       "default-address-pools": [{"base": "172.17.0.0/16", "size": 24}]
   }

2. dockerd starts and creates the default bridge (docker0) at 172.17.0.0/24 (first /24 from the pool).
3. The task runs docker compose up, which creates mint-workspace_default at 172.17.1.0/24 (next /24 from the same pool).
4. This Docker state (including /var/lib/docker/network/files/local-kv.db) is cached as a Docker layer.
5. A subsequent task restores this cached layer, including the stale network database.
6. dockerd starts and tries to create a new docker0 bridge, but the IPAM pool (172.17.0.0/16) already has 172.17.1.0/24 allocated to br-a20d7f349c72 from the restored state. Docker detects the overlap and refuses to start.

The TypeScript agent does not write a daemon.json — Docker uses its default pools which allocate separate /16 blocks per network (e.g., 172.17.0.0/16 for docker0, 172.18.0.0/16 for the first user network), but this can still fail when stale state is restored with networks in the same range.

Fix

Remove /var/lib/docker/network/ from the prepared Docker directory after layer restoration but before the container starts. The cached network metadata (bridge IPs, interface names, IPAM allocations) is invalid in a new network namespace. Docker recreates this directory and the default bridge cleanly on startup. Docker images, layers, and image metadata elsewhere in /var/lib/docker/ are unaffected.