Notes from building ETCD / Confd / Haproxy autoconfiguration environment

README.md

The following documents a trial of using etcd, and confd to automatically configure a haproxy load balancer.

It is built using a combination of blogs, resources and experimentation, but provides a rough template of the approach that would allow a fully featured balancer to be configured from etcd keyvalues.

TODO

• Include systemd sidekick unit to automatically register the key's into ETCD based on a docker service (for example) being started
• A more complete haproxy template that builds a valid, complex haproxy that would be capable of dealing with Future PLC's balancer requirements

ETCD (v3) Installation and Setup

Follow instructions from https://github.com/coreos/etcd/releases then...

1. mv /tmp/test-etcd/etcd* /usr/bin/
2. Add etcd.system script shown above to correct location and follow instructions in the file
3. journal -xe -u etcd should show logs that etcd is started
4. Confirm with etcdctl cluster-health && etcdctl member list

ConfD Installation and Setup

wget https://github.com/kelseyhightower/confd/releases/download/v0.11.0/confd-0.11.0-linux-amd64
mv confd-0.11.0-linux-amd64 /usr/bin/confd

The follow the quick start guide to validate things are working; https://github.com/kelseyhightower/confd/blob/master/docs/quick-start-guide.md

HAProxy Templates and Usage

(Based on https://github.com/Ventures/haproxy-confd )

Create a new confd configuration based on the haproxy.toml and haproxy.tmpl

Configure the ETCD keys using the following basic example;

etcdctl mkdir "/haproxy-etcd/services"
etcdctl set "/haproxy-etcd/services/etcd/domain" "etcd.lan"
etcdctl set "/haproxy-etcd/services/etcd/port" "80"

And run confd with the following for a one off test case

confd -onetime -backend etcd -node http://etcd.lan:2379 -log-level=debug

And use this to get confd to keep a watch on the etcd keys and rebuild the config if any of the values change;

confd -backend etcd -node http://etcd.lan:2379 -log-level=debug -watch=true

* WARNING: * - Running with watch on a production system might be a bad idea if all your balancers spontaniously reconfigure themselves concurrently.

Example output

confd -onetime -backend etcd -node http://etcd.lan:2379 -log-level=debug

2016-11-21T21:48:23Z haproxy confd[7569]: INFO Backend set to etcd
2016-11-21T21:48:23Z haproxy confd[7569]: INFO Starting confd
2016-11-21T21:48:23Z haproxy confd[7569]: INFO Backend nodes set to http://etcd.lan:2379
2016-11-21T21:48:23Z haproxy confd[7569]: DEBUG Loading template resources from confdir /etc/confd
2016-11-21T21:48:23Z haproxy confd[7569]: DEBUG Loading template resource from /etc/confd/conf.d/haproxy.toml
2016-11-21T21:48:23Z haproxy confd[7569]: DEBUG Retrieving keys from store
2016-11-21T21:48:23Z haproxy confd[7569]: DEBUG Key prefix set to /haproxy-etcd
2016-11-21T21:48:23Z haproxy confd[7569]: DEBUG Using source template /etc/confd/templates/haproxy.tmpl
2016-11-21T21:48:23Z haproxy confd[7569]: DEBUG Compiling source template /etc/confd/templates/haproxy.tmpl
2016-11-21T21:48:23Z haproxy confd[7569]: DEBUG Comparing candidate config to /etc/haproxy/haproxy.conf
2016-11-21T21:48:23Z haproxy confd[7569]: INFO Target config /etc/haproxy/haproxy.conf out of sync
2016-11-21T21:48:23Z haproxy confd[7569]: DEBUG Running /usr/sbin/haproxy -c -f /etc/haproxy/.haproxy.conf670147617
2016-11-21T21:48:23Z haproxy confd[7569]: DEBUG "Configuration file is valid\n"
2016-11-21T21:48:23Z haproxy confd[7569]: DEBUG Overwriting target config /etc/haproxy/haproxy.conf
2016-11-21T21:48:23Z haproxy confd[7569]: DEBUG Running /usr/sbin/service haproxy reload
2016-11-21T21:48:23Z haproxy confd[7569]: DEBUG ""
2016-11-21T21:48:23Z haproxy confd[7569]: INFO Target config /etc/haproxy/haproxy.conf has been updated

Registrator - Watching Docker containers

Run this on your docker server and it'll ping back to etcd when services start...

docker run -d \
>     --name=registrator \
>     --net=host \
>     --volume=/var/run/docker.sock:/tmp/docker.sock \
>     gliderlabs/registrator:latest \
>     etcd://etcd:2379/manager

Then run redis (for kicks and giggles)

docker run -d -P --name=redis redis

Then look at etcd

 etcdctl ls /manager/redis
/manager/redis/manager:redis:6379

Thats right! Docker container started, registrator told ETCD and etcd can now create configuration for the backend redis instances automatically.

Magic!

etcd.service

#/etc/systemd/service/etcd.service
# Based on https://github.com/coreos/etcd/blob/master/contrib/systemd/etcd.service
#
# Usage:
#   Update Advertise IP's in Environment below
#   useradd etcd -Mr
#   mkdir /var/lib/etcd
#   chown etcd. /var/lib/etcd
#   systemctl daemon-reload
#   service etcd start
#
[Unit]
Description=etcd key-value store
Documentation=https://github.com/coreos/etcd
After=network.target

[Service]
User=etcd
Type=notify
Environment=ETCD_DATA_DIR=/var/lib/etcd
Environment=ETCD_NAME=%m
Environment=ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
Environment=ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
Environment=ETCD_ADVERTISE_CLIENT_URLS=http://%H:2379
ExecStart=/usr/bin/etcd
Restart=always
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target

haproxy.tmpl

# /etc/confd/templates/haproxy.tmpl

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
	ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

## End of HAProxy Defaults Preamble


# Hardcoded for the moment
listen http-in
        bind *:80


# For each 'service' defined under the haproxy-etcd application
# create a new backend element
{{range $service := ls "/services"}}
backend {{$service}}
    option                  forwardfor
    {{$key := printf "/services/%s/balance" $service}}  {{if exists $key}}
    balance                 {{getv $key}}
    {{else}}
    balance                 roundrobin
    {{end}}
    server {{$service}} {{printf "/services/%s/domain" $service | getv}}:{{printf "/services/%s/port" $service | getv}} check
{{end}}

haproxy.toml

# /etc/confd/confd/haproxy.toml

[template]
prefix = "/haproxy-etcd"
src = "haproxy.tmpl"
dest = "/etc/haproxy/haproxy.conf"
owner = "haproxy"
mode = "0644"
keys = [
  "/services",
]
check_cmd = "/usr/sbin/haproxy -c -f {{.src}}"
reload_cmd = "/usr/sbin/service haproxy reload"