stress testing the creation of a 1000 docker containers, comes across file descriptor and iptable errors

stress-docker-run

[] ubuntu@~$ sudo docker version
Client version: 0.6.3
Go version (client): go1.1.2
Git commit (client): b0a49a3
Server version: 0.6.3
Git commit (server): b0a49a3
Go version (server): go1.1.2
Last stable version: 0.6.4, please update docker

[] ubuntu@~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15962
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15962
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

# hint from https://groups.google.com/forum/#!topic/docker-user/uWiC5R9CqEY
[] ubuntu@~$ cat /proc/sys/fs/inotify/max_user_watches
8192

[] ubuntu@~$ cat /stress.sh
#!/bin/bash
echo "Creating 1000 containers"
df -h .
for i in {1..1000}; do
id=`sudo docker run -d test/ssh /usr/sbin/sshd`
#echo "$i => $id" -n
echo -n "."
done
df -h .

[] ubuntu@~$ /stress.sh
Creating 1000 containers
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         40G  1.7G   36G   5% /
......................................................................................................................................................................................................................................................................................................................................................................................2013/10/18 12:57:45 Error: Error starting container da83e4a474ea: pipe2: too many open files
.2013/10/18 12:57:45 Error: Error starting container e13e6a60c69e: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50650 ! -i docker0 -j DNAT --to-destination 172.17.5.219:22
..2013/10/18 12:57:45 Error: Error starting container 41e1c9090d64: pipe2: too many open files
.2013/10/18 12:57:45 Error: Error starting container 3c2390c7ae54: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50653 ! -i docker0 -j DNAT --to-destination 172.17.5.222:22
.2013/10/18 12:57:45 Error: Error starting container 79b3976fc31c: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50654 ! -i docker0 -j DNAT --to-destination 172.17.5.223:22
.2013/10/18 12:57:46 Error: Error starting container 8a418403d944: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50655 ! -i docker0 -j DNAT --to-destination 172.17.5.224:22
.2013/10/18 12:57:46 Error: Error starting container 2112e187fe7f: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50656 ! -i docker0 -j DNAT --to-destination 172.17.5.225:22
.2013/10/18 12:57:46 Error: Error starting container ce5139c6c081: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50657 ! -i docker0 -j DNAT --to-destination 172.17.5.226:22
.2013/10/18 12:57:46 Error: Error starting container 85ee1f6c7922: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50658 ! -i docker0 -j DNAT --to-destination 172.17.5.227:22
.2013/10/18 12:57:46 Error: Error starting container 3e3dc60dee65: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50659 ! -i docker0 -j DNAT --to-destination 172.17.5.228:22
.2013/10/18 12:57:46 Error: Error starting container 1ec26be9e8a2: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50660 ! -i docker0 -j DNAT --to-destination 172.17.5.229:22
..................2013/10/18 12:57:49 Error: Error starting container 9d82210e63fc: pipe2: too many open files
...2013/10/18 12:57:49 Error: Error starting container f0c14d1c99f8: fork/exec /usr/bin/lxc-start: too many open files
.2013/10/18 12:57:49 Error: Error starting container a8afb1837e38: pipe2: too many open files
.2013/10/18 12:57:49 Error: Error starting container cf198bd6a0cb: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50683 ! -i docker0 -j DNAT --to-destination 172.17.5.252:22
.2013/10/18 12:57:49 unexpected EOF
.2013/10/18 12:57:49 unexpected EOF
.2013/10/18 12:57:49 Error: Error starting container bd0cfc17d22c: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50686 ! -i docker0 -j DNAT --to-destination 172.17.5.255:22
.2013/10/18 12:57:49 unexpected EOF
.....................2013/10/18 12:57:52 Error: Error starting container 40e1599d74bf: pipe2: too many open files
.2013/10/18 12:57:52 Error: Error starting container a3acd0dbdc54: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50709 ! -i docker0 -j DNAT --to-destination 172.17.6.22:22
.2013/10/18 12:57:53 Error: Error starting container 1dce00bc8f72: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50710 ! -i docker0 -j DNAT --to-destination 172.17.6.23:22
.2013/10/18 12:57:53 read unix /var/run/docker.sock: connection reset by peer
....2013/10/18 12:57:53 Error: Error starting container 10d95b1a1fe2: fork/exec /usr/bin/lxc-start: too many open files
..2013/10/18 12:57:53 Error: Error starting container 246f6d3eb751: pipe2: too many open files
.2013/10/18 12:57:53 unexpected EOF
.2013/10/18 12:57:53 unexpected EOF
.2013/10/18 12:57:53 Error: Error starting container 1b68396e5c89: pipe2: too many open files
.2013/10/18 12:57:54 unexpected EOF
.2013/10/18 12:57:54 Error: Error starting container d86ea259a1ea: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50721 ! -i docker0 -j DNAT --to-destination 172.17.6.34:22
.2013/10/18 12:57:54 unexpected EOF
.2013/10/18 12:57:54 unexpected EOF
.2013/10/18 12:57:54 unexpected EOF
.2013/10/18 12:57:54 Error: Error starting container 719321a68b47: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50725 ! -i docker0 -j DNAT --to-destination 172.17.6.38:22
.2013/10/18 12:57:54 Error: Error starting container 08eb459f2b89: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50726 ! -i docker0 -j DNAT --to-destination 172.17.6.39:22
.2013/10/18 12:57:54 Error: Error starting container b8f78a7cb7fd: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50727 ! -i docker0 -j DNAT --to-destination 172.17.6.40:22
.2013/10/18 12:57:54 Error: Error starting container 63f0a3057b0c: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50728 ! -i docker0 -j DNAT --to-destination 172.17.6.41:22
.2013/10/18 12:57:54 unexpected EOF
.2013/10/18 12:57:54 Error: Error starting container 8ac2aeab8b11: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50730 ! -i docker0 -j DNAT --to-destination 172.17.6.43:22
....2013/10/18 12:57:55 Error: Error starting container 10da41d68cba: pipe2: too many open files
.2013/10/18 12:57:55 Error: Error starting container f029e33f23f9: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50735 ! -i docker0 -j DNAT --to-destination 172.17.6.48:22
.2013/10/18 12:57:55 Error: Error starting container 54d7d0535c17: pipe2: too many open files
.2013/10/18 12:57:55 Error: Error starting container c1a50e08dcb8: fork/exec /usr/bin/lxc-start: too many open files
........2013/10/18 12:57:56 Error: Error starting container ccffe2fdf595: pipe2: too many open files
.2013/10/18 12:57:56 Error: Error starting container f5595aeec957: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50746 ! -i docker0 -j DNAT --to-destination 172.17.6.59:22
.2013/10/18 12:57:56 Error: Error starting container ce145463a442: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50747 ! -i docker0 -j DNAT --to-destination 172.17.6.60:22
.2013/10/18 12:57:57 Error: Error starting container 04567cd2d2f3: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50748 ! -i docker0 -j DNAT --to-destination 172.17.6.61:22
.2013/10/18 12:57:57 Error: Error starting container 57d217a3bb82: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50749 ! -i docker0 -j DNAT --to-destination 172.17.6.62:22
.2013/10/18 12:57:57 Error: Error starting container a2b8933531e5: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50750 ! -i docker0 -j DNAT --to-destination 172.17.6.63:22
.2013/10/18 12:57:57 unexpected EOF
.2013/10/18 12:57:57 Error: Error starting container 1af26c868a0c: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50752 ! -i docker0 -j DNAT --to-destination 172.17.6.65:22
.2013/10/18 12:57:57 unexpected EOF
.2013/10/18 12:57:57 unexpected EOF
.2013/10/18 12:57:57 unexpected EOF
.2013/10/18 12:57:57 Error: Error starting container 849223b54aa3: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50756 ! -i docker0 -j DNAT --to-destination 172.17.6.69:22
.2013/10/18 12:57:57 Error: Error starting container 7003a2e9d162: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50757 ! -i docker0 -j DNAT --to-destination 172.17.6.70:22
.2013/10/18 12:57:57 Error: Error starting container d743ecc0754d: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50758 ! -i docker0 -j DNAT --to-destination 172.17.6.71:22
.2013/10/18 12:57:57 Error: Error starting container 3afa52ed17a5: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50759 ! -i docker0 -j DNAT --to-destination 172.17.6.72:22
.2013/10/18 12:57:57 Error: Error starting container f3d41acce65f: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50760 ! -i docker0 -j DNAT --to-destination 172.17.6.73:22
.2013/10/18 12:57:57 Error: Error starting container 8b710126c54b: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50761 ! -i docker0 -j DNAT --to-destination 172.17.6.74:22
.2013/10/18 12:57:58 unexpected EOF
.2013/10/18 12:57:58 Error: Error starting container a923d8343807: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50763 ! -i docker0 -j DNAT --to-destination 172.17.6.76:22
.2013/10/18 12:57:58 unexpected EOF
.2013/10/18 12:57:58 Error: Error starting container 8f5b446472ea: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50765 ! -i docker0 -j DNAT --to-destination 172.17.6.78:22
.2013/10/18 12:57:58 unexpected EOF
.2013/10/18 12:57:58 Error: Error starting container 42e6b095787e: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 50767 ! -i docker0 -j DNAT --to-destination 172.17.6.80:22
.2013/10/18 12:57:58 unexpected EOF
.2013/10/18 12:57:58 unexpected EOF
...........................................................................................................................................2013/10/18 12:58:22 unexpected EOF
..................................................................................................................................................2013/10/18 12:58:45 Error: Error starting container 59ad04fc96e1: iptables failed: iptables -t nat -A DOCKER -p tcp --dport 51054 ! -i docker0 -j DNAT --to-destination 172.17.7.111:22
............................................................................................................................................................................................................................Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         40G  1.8G   36G   5% /


#
# here's how test/ssh was created
#

[] ubuntu@~$ cat Dockerfile 
FROM base
EXPOSE 22

RUN sudo apt-get update
RUN dpkg-divert --local --rename --add /sbin/initctl && ln -s /bin/true /sbin/initctl

RUN sudo apt-get install -q -y openssh-server
RUN mkdir /var/run/sshd

[] ubuntu@~$ sudo docker build -t test/ssh .