Registrator / SkyDNS for CoreOS / Deis cluster

_README.md

Registrator and SkyDNS

We use progrium/registrator and yaronr/skydns (SkyDNS2) to publish information about Docker containers to DNS via A and SRV records. All nodes runs skydns and registrator, and first three nodes are inserted as NS-s into Route53 DNS for services.cluster-name.domain.io. Note, v4 registrator must be used until registrator/124 is resolved.

$ host -t srv mysql-1.services.deis.r53.acp.io deis-6-1.eu.r53.acp.io
Using domain server:
Name: deis-6-1.eu.r53.acp.io
Address: 54.171.239.227#53
Aliases: 

mysql-1.services.deis.r53.acp.io has SRV record 10 100 49153 ec2-54-171-239-227.eu-west-1.compute.amazonaws.com.

10 is priority, 100 is weight, 49153 is MySQL port. The DNS name is either the Docker image (default) or could be set via SERVICE_NAME env var, see mysql-1.service.

You can ask for internal per-container IP of Flannel network by using internal zone:

$ host -t a mathics.internal.services.deis-7.containers.r53.acp.io deis-7-1.eu.r53.acp.io
Using domain server:
Name: deis-7-1.eu.r53.acp.io
Address: 54.154.28.131#53
Aliases: 

mathics.internal.services.deis-7.containers.r53.acp.io has address 10.42.47.29
mathics.internal.services.deis-7.containers.r53.acp.io has address 10.42.47.30
mathics.internal.services.deis-7.containers.r53.acp.io has address 10.42.12.21

Thus communicating with services deployed via Fleet/etc. is possible on well-known ports, without digging into SRV records or using Kubernetes env vars. Also note, an A query to *.internal.services will return all IP addresses handling the service, while a query to *.services will return a CNAME to *.compute.amazonaws.com in round-robin fashion, which in turn will resolve to VPC IP address when queried inside the AWS, and public IP addresses outside of it. Asking for SRV record will always return all hosts:

$ host -t srv skydns-53.services.deis-8.containers.r53.acp.io ec2-54-154-44-145.eu-west-1.compute.amazonaws.com
Using domain server:
Name: ec2-54-154-44-145.eu-west-1.compute.amazonaws.com
Address: 54.154.44.145#53
Aliases: 

skydns-53.services.deis-8.containers.r53.acp.io has SRV record 10 33 53 ec2-54-154-44-36.eu-west-1.compute.amazonaws.com.
skydns-53.services.deis-8.containers.r53.acp.io has SRV record 10 33 53 ec2-54-154-44-145.eu-west-1.compute.amazonaws.com.
skydns-53.services.deis-8.containers.r53.acp.io has SRV record 10 33 53 ec2-54-154-44-144.eu-west-1.compute.amazonaws.com.

Note the internal zone SRV format is different:

$ host -t srv skydns-53.internal.services.deis-8.containers.r53.acp.io ec2-54-154-44-145.eu-west-1.compute.amazonaws.com
;; Truncated, retrying in TCP mode.
Using domain server:
Name: ec2-54-154-44-145.eu-west-1.compute.amazonaws.com
Address: 54.154.44.145#53
Aliases: 

skydns-53.internal.services.deis-8.containers.r53.acp.io has SRV record 10 33 53 ip-10-21-1-134.eu-west-1.compute.internal:skydns:53.skydns-53.internal.services.deis-8.containers.r53.acp.io.
skydns-53.internal.services.deis-8.containers.r53.acp.io has SRV record 10 33 53 ip-10-21-2-103.eu-west-1.compute.internal:skydns:53.skydns-53.internal.services.deis-8.containers.r53.acp.io.
skydns-53.internal.services.deis-8.containers.r53.acp.io has SRV record 10 33 53 ip-10-21-2-102.eu-west-1.compute.internal:skydns:53.skydns-53.internal.services.deis-8.containers.r53.acp.io.

docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.io
After=docker.socket
Requires=docker.socket
Requires=flannel.service
After=flannel.service
Requires=var-lib-docker.mount
After=var-lib-docker.mount

[Service]
Environment="TMPDIR=/var/tmp/"
EnvironmentFile=/run/flannel/subnet.env
EnvironmentFile=-/run/docker_opts.env
ExecStartPre=/bin/mount --make-rprivate /
# no-COW flag will propagate down the tree as directories are created by Docker - this is only important on first run
ExecStartPre=/usr/bin/chattr +C /var/lib/docker
LimitNOFILE=1048576
LimitNPROC=1048576
ExecStart=/bin/sh -c 'exec /usr/bin/docker --daemon --storage-driver=btrfs --host=fd:// --bip=${FLANNEL_SUBNET} --mtu=${FLANNEL_MTU} $DOCKER_OPTS'
Restart=always

flannel.service

[Unit]
Description=Flannel overlay network
Requires=etcd.service
After=etcd.service
Requires=network-online.target
After=network-online.target

[Service]
TimeoutStartSec=300
ExecStartPre=/bin/mkdir -p /opt/bin
ExecStartPre=/bin/sh -xc 'f=/opt/bin/flanneld; test -x $f || wget --retry-connrefused -O $f my-release-bucket.s3-eu-west-1.amazonaws.com/bin/flanneld && chmod +x $f'
ExecStartPre=/bin/sh -c 'i=0; while sleep 1; do i=$((i+1)); test $i -gt 20 && exit 1; etcdctl mk /coreos.com/network/config {\\"Network\\":\\"10.42.0.0/16\\"}; test $? -eq 0 -o $? -eq 4 && exit 0; done'
ExecStart=/opt/bin/flanneld
Restart=always

[Install]
RequiredBy=docker.service

mysql-1.service

[Unit]
Description=MySQL-1
Requires=docker.service
After=docker.service

[Service]
TimeoutStartSec=300
ExecStartPre=/usr/bin/docker pull mysql:5.7
ExecStart=/bin/sh -c '/usr/bin/docker start -a mysql-1 || exec docker run --rm --name mysql-1 -p 3306 -v /var/lib/deis/store/mysql-1:/var/lib/mysql -e MYSQL_ROOT_PASSWORD=qwerty -e MYSQL_USER=springboot -e MYSQL_PASSWORD=qwerty -e MYSQL_DATABASE=tasks -e SERVICE_NAME=mysql-1 mysql:5.7'
ExecStop=/usr/bin/docker stop mysql-1
ExecStop=-/usr/bin/docker rm mysql-1
Restart=always

registrator-internal.service

[Unit]
Description=Docker Registrator - flannel network
Requires=docker.service
Wants=docker.service

[Service]
TimeoutStartSec=300
#ExecStartPre=-/usr/bin/docker kill registrator-internal
#ExecStartPre=-/usr/bin/docker rm registrator-internal
ExecStartPre=/usr/bin/docker pull docker-registry.domain.io:5000/progrium/registrator:v4
ExecStart=/bin/sh -c '/usr/bin/docker start -a registrator-internal || exec docker run --name registrator-internal -h $HOSTNAME -v /var/run/docker.sock:/tmp/docker.sock docker-registry.domain.io:5000/progrium/registrator:v4 -internal -ttl 30 -ttl-refresh 20 skydns2://$(ip addr show dev docker0 |grep "inet " |sed -re "s/.+ ([0-9\\.]+).+/\\1/g"):4001/internal.services.{{domain}}'
ExecStop=/usr/bin/docker stop registrator-internal
ExecStop=/usr/bin/docker rm registrator-internal
Restart=always

[X-Fleet]
Global=true

registrator.service

[Unit]
Description=Docker Registrator - host network
Requires=docker.service
Wants=docker.service

[Service]
TimeoutStartSec=300
#ExecStartPre=-/usr/bin/docker kill registrator
#ExecStartPre=-/usr/bin/docker rm registrator
ExecStartPre=/usr/bin/docker pull docker-registry.domain.io:5000/progrium/registrator:v4
ExecStart=/bin/sh -c '/usr/bin/docker start -a registrator || exec docker run --name registrator -h $HOSTNAME -v /var/run/docker.sock:/tmp/docker.sock docker-registry.domain.io:5000/progrium/registrator:v4 -ip $(curl -sS http://169.254.169.254/latest/meta-data/public-hostname) -ttl 30 -ttl-refresh 20 skydns2://$(ip addr show dev docker0 |grep "inet " |sed -re "s/.+ ([0-9\\.]+).+/\\1/g"):4001/services.{{domain}}'
ExecStop=/usr/bin/docker stop registrator
ExecStop=/usr/bin/docker rm registrator
Restart=always

[X-Fleet]
Global=true

skydns.service

[Unit]
Description=SkyDNS
Wants=docker.service
After=docker.service

[Service]
TimeoutStartSec=300
#ExecStartPre=-/usr/bin/docker kill skydns
#ExecStartPre=-/usr/bin/docker rm skydns
ExecStartPre=/usr/bin/docker pull yaronr/skydns:latest
ExecStartPre=-/usr/bin/etcdctl mk /skydns/config '{"dns_addr":"0.0.0.0:53", "domain": "services.{{domain}}", "ttl":30, "nameservers": ["10.21.0.2:53"]}'
ExecStart=/bin/sh -c '/usr/bin/docker start -a skydns || exec docker run --name skydns -p 53:53/udp -p 53:53/tcp -e ETCD_MACHINES=http://$(ip addr show dev docker0 |grep "inet " |sed -re "s/.+ ([0-9\\.]+).+/\\1/g"):4001/ yaronr/skydns'
ExecStop=/usr/bin/docker stop skydns
ExecStop=/usr/bin/docker rm skydns
Restart=always

[X-Fleet]
Global=true