`traefik`, `acme-dns`, and `watchtower` as Docker web application helpers

!docker-webapps.md

This setup configures traefik, acme-dns, and watchtower as system wide Docker services to ease the deployment of docker web services.

Please read the content for more information.

content.md

Docker Web Services Helpers

ARCHIVED: While this works fine, I have moved on from this style of deployment and wont be using this for any new setups.

This config defines three independent helper services that are useful when hosting (multiple) web applications on a docker host:

• Traefik: An "edge router". Here used to route TLS connections to the individual web apps, based on the SNI host name, without decrypting the connection.
• ACME-DNS: Limited DNS server, which allows usage of The DNS challenge for Let's Encrypt wildcard certificates with any domain or registrar, and in this setup also without explicit credential management.
• Watchtower: Container update service, which queries the Docker registry for newer image versions, then downloads the images and redeploys the updated containers.

The script below can be used to set up any subset of these three services (the services to be started can be selected after the setup). To get started, review and adjust these variables and set them in a root bash (most should be sensible defaults):

# { . <(cat << "#EOF"
#!/usr/bin/env bash

appsRootDir=/app/ # Host directory for this compose project, with tailing '/'. TODO: /srv might actually be the FHS compliant place for this
appsRootZfsDataset=$(mount | grep -oP ".*(?= on $(grep -oP '.*[^/]' <<< "$appsRootDir") type zfs )")/ # Optional, ZFS filesystem mounted at `appsRootDir`. If supplied, some sub dirs will be put on individual child datasets of this.

appName=webapps # Name of the project, and its top level directory/dataset.
externalIPv4=$(ip route get 1.1.1.1 | grep -oP 'src \K\S+') # IPv4 published as DNS server.
externalIPv6=$(ip route get 1111:1111:1111:: | grep -oP 'src \K\S+') # Optional IPv6 published as DNS server.
bind53IPv4=$externalIPv4 # IPv4 the DNS server actually listens on.
bind53IPv6=$externalIPv6 # IPv6 the DNS server actually listens on.
acmeDnsDomain=acme.$(hostname) # Subdomain used for the ACME-DNS server.
#EOF
); }

If ACME-DNS is to be used, and assuming those variables above, set these DNS entries with your current provider for domain:

${acmeDnsDomain}  A      ${externalIPv4}
${acmeDnsDomain}  AAAA   ${externalIPv6} # optional
${acmeDnsDomain}  NS     ${acmeDnsDomain}

Then, with the above variables set, run:

# { (tempScript=$(mktemp); . <(tee ${tempScript} << "#EOF" # copy from after the first #
#!/usr/bin/env bash
set -eux

## Functions

function createAppDataset { # 1: perms, 2: owner, 3: relPath
    if [[ "$2" == '' || "$2" == ':' ]]; then return 1; fi
    if [[ -e "${appsRootDir}${3}" ]]; then return; fi
    mkdir -p ${appsRootDir}${3};
    if [[ "${appsRootZfsDataset}" ]]; then
        chmod 000 ${appsRootDir}${3}; chattr +i ${appsRootDir}${3}
        zfs create ${appsRootZfsDataset}${3}
    fi
    chmod ${1} ${appsRootDir}${3}; chown ${2} ${appsRootDir}${3};
}
function getUid {( # 1: app, 2: user? => "$uid:$gid\t$uid\t$gid"
    set -ex
    local image=$(docker-compose images -q "$1")
    if [[ ! "$image" ]]; then
        docker-compose up --no-start "$1" 1>&2
        image=$(docker-compose images -q "$1")
    fi
    local rand=$(</dev/urandom tr -dc A-Za-z0-9 | head -c 48)
    mkdir /tmp/$rand/; chmod 777 /tmp/$rand/
    user=(); if [[ ${2:-} ]]; then user=("--user=$2"); fi
    docker run --rm --volume=/tmp/$rand/:/tmp/ "${user[@]}" --entrypoint /bin/sh "$image" -c 'touch /tmp/gid_test' 1>&2
    gid="$(stat -c '%g' /tmp/$rand/gid_test)"
    uid="$(stat -c '%u' /tmp/$rand/gid_test)"
    rm -f /tmp/$rand/gid_test; rmdir /tmp/$rand/
    printf '%s\t%s\t%s' "$uid:$gid" "$uid" "$gid"
)}


## Preparation/.env

appRoot=${appsRootDir}${appName}
createAppDataset 755 root ${appName}
createAppDataset 755 root ${appName}/config

# if run from CLI, create backup of script and config
if [[ "${tempScript:-}" && -e "${tempScript}" ]]; then
    cat "${tempScript}" > ${appRoot}/config/setup.sh; rm "${tempScript}"
fi

# env vars for this setup and docker-compose
if [[ ! -e "${appRoot}/config/.env" ]]; then
    touch ${appRoot}/config/.env; chmod 640 ${appRoot}/config/.env
    envFile=${appRoot}/config/.env; : "Saving configuration in ${appRoot}/config/.env"
else
    envFile=/dev/null # just to make sure all the required vars are still set
fi
cat << EOC > ${envFile}
# Do NOT quote any of the values in this file!
appsRootDir=${appsRootDir}
appsRootZfsDataset=${appsRootZfsDataset}

appName=${appName}
appRoot=${appRoot}
externalIPv4=${externalIPv4}
externalIPv6=${externalIPv6}
bind53IPv4=${bind53IPv4}
bind53IPv6=${bind53IPv6}
acmeDnsDomain=${acmeDnsDomain}

COMPOSE_FILE=docker-compose.yml:docker-compose.d/traefik.yml:docker-compose.d/acme-dns.yml:docker-compose.d/watchtower.yml:docker-compose.override.yml
COMPOSE_PATH_SEPARATOR=:
COMPOSE_PROJECT_NAME=${appName}
EOC


## docker-compose

cat << 'EOC' > ${appRoot}/config/docker-compose.yml
version: '3.5'

services: { } # see ./docker-compose.d/

volumes:
    empty: # dummy to avoid spamming anonymous volumes
EOC

mkdir -p ${appRoot}/config/docker-compose.d/
cat << 'EOC' > ${appRoot}/config/docker-compose.d/traefik.yml
version: '3.5'

services:
    traefik:
        image: 'traefik:v2.2'
        command:
            #- --log.level=DEBUG
            - --api.insecure=true # expose API without authentication
            - --providers.docker=true
            - --providers.docker.exposedbydefault=false
            - --entrypoints.web.address=:10080
            - --entrypoints.web.http.redirections.entryPoint.to=:443
            - --entrypoints.web.http.redirections.entryPoint.scheme=https
            - --entrypoints.web.http.redirections.entryPoint.permanent=true
            - --entrypoints.websecure.address=:10443
        networks: [ traefik ]
        ports:
            - '80:10080'
            - '443:10443'
            #- '127.0.0.1:8080:8080' # expose (unauthenticated) API only on localhost
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock:ro
        restart: unless-stopped
        labels:
            com.centurylinklabs.watchtower.enable: 'true'
        userns_mode: host # TODO: this sucks

networks:
    traefik: { name: traefik } # create a network with a system wide fixed name, that other project's containers can be joined into
EOC
cat << 'EOC' > ${appRoot}/config/docker-compose.d/acme-dns.yml
version: '3.5'

services:
    acme-dns:
        image: 'joohoi/acme-dns:v0.8'
        entrypoint:
            - /bin/sh
            - -c
            - |
                set -e; sed \
                    -e 's/$${CONFIG_DOMAIN}/${acmeDnsDomain}/g' \
                    -e 's/$${CONFIG_IPv4}/${externalIPv4}/g' \
                    -e 's/$${CONFIG_IPv6}/${externalIPv6}/g' \
                < /root/config.cfg.template \
                | /root/acme-dns -c /dev/stdin
        networks: [ acme-dns ]
        expose: [ 8080 ]
        ports:
            - '${bind53IPv4}:53:53/tcp'
            - '${bind53IPv4}:53:53/udp'
            - '${bind53IPv6}:53:53/tcp'
            - '${bind53IPv6}:53:53/udp'
        volumes:
            - ${appRoot}/acme-dns:/var/lib/acme-dns
            - ${appRoot}/config/acme-dns.toml:/root/config.cfg.template:ro
            - empty:/etc/acme-dns:ro
        restart: unless-stopped
        labels:
            com.centurylinklabs.watchtower.enable: 'true'

networks:
    acme-dns: { name: acme-dns }
EOC
cat << 'EOC' > ${appRoot}/config/docker-compose.d/watchtower.yml
version: '3.5'

services:
    # allow apps (host system wide) to set `labels: { com.centurylinklabs.watchtower.enable: 'true' }` to opt in to auto updates
    watchtower:
        userns_mode: host # access to the docker socket effectively grands root privileges anyway
        image: 'containrrr/watchtower:latest'
        command: --schedule "0 21 4 * * *" --cleanup --stop-timeout 30s --label-enable
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - /etc/timezone:/etc/timezone:ro
        restart: unless-stopped
        labels:
            com.centurylinklabs.watchtower.enable: 'true'
EOC
test -e ${appRoot}/config/docker-compose.override.yml || printf "version: '3.5'\n\nservices: { }\n" > ${appRoot}/config/docker-compose.override.yml # for site specific stuff


## acme-dns

(cd ${appRoot}/config/; createAppDataset g+rwx,o=r,+t :$(COMPOSE_FILE=docker-compose.yml:docker-compose.d/acme-dns.yml getUid acme-dns | cut -f3) ${appName}/acme-dns)

cat << 'EOC' > ${appRoot}/config/acme-dns.toml
[general]
listen = "0.0.0.0:53" # all interfaces in the container
protocol = "both" # UDP and TCP on IPv4 and IPv6
domain = "${CONFIG_DOMAIN}"
nsname = "${CONFIG_DOMAIN}"
nsadmin = "admin.${CONFIG_DOMAIN}" # SOA RNAME
records = [
    "${CONFIG_DOMAIN}. A ${CONFIG_IPv4}",
    "${CONFIG_DOMAIN}. AAAA ${CONFIG_IPv6}",
    "${CONFIG_DOMAIN}. NS ${CONFIG_DOMAIN}.",
]
debug = false

[database]
engine = "sqlite3"
connection = "/var/lib/acme-dns/acme-dns.db"

[api]
ip = "0.0.0.0"
disable_registration = false
port = "80"
tls = "none"
corsorigins = [ "*" ]
use_header = false

[logconfig]
# logging level: "error", "warning", "info" or "debug"
loglevel = "debug"
logtype = "stdout"
logformat = "text"
EOC

if which ufw >/dev/null; then # if using UFW, allow DNS in
    ufw allow 53/tcp comment acme-dns; ufw allow 53/udp comment acme-dns
fi


## traefik

# (See TODO below. This was an attempt to create a user outside the container that has the same UID as the traefik user in the container, and is in the docker group. It still couldn't read the socket.)
## add a user matching that in the traefik container and add it to the docker group
#uid_traefik_root_2=$(getUid traefik root | cut -f 2)
#adduser traefik --system --gecos "" --disabled-password --no-create-home || true
#usermod -u $((uid_traefik_root_2 + 28274)) traefik
#usermod -aG docker traefik
# compose file snippet:
#    entrypoint:
#        - /bin/sh
#        - -c
#        - |
#            if ! id -u traefik 1>/dev/null 2>/dev/null ; then
#                adduser traefik -g "" -u 28274 -D
#            fi
#            su traefik -c 'set -x; source /entrypoint.sh' -- "$$0" "$$@"

if which ufw >/dev/null; then # if using UFW, allow HTTP(S) in
    ufw allow http comment 'to https'; ufw allow https comment traefik
fi

#EOF
)); }

If any of the three services is not required on the host, editor ${appsRootDir}${appName}/config/.env and remove the corresponding entry from the COMPOSE_FILE list. Now all that's left to do is to call docker-compose up -d (in ${appRoot}/config/), and then to add the actual web apps. To re-apply the configuration later, cd ${appRoot}/config/, source .env, and either paste the script again, or source setup.sh the saved copy.

TODO

• traefik does not support wildcards or regexps in the HostSNI(). envoy could be an alternative.
• traefik runs as root, but since it only requires read access to the docker socket, it should be possible to run it as non-root (in the docker group).

ARCHIVED: While this works fine, I have moved on from this style of deployment and wont be using this for any new setups.

May I ask.. what have you moved on towards? :)

Sure! I am using the NixOS Linux distribution now. While one can use Docker(-Compose) (and thus this setup) on NixOS, NixOS actually provides some different, more fine-grained and system-integrated ways to run isolated and/or containerized applications:

• acme-dns and traefik can quite easily be replaces by NixOS's security.acme and services.nginx options
• watchtower is not needed, as the applications/containers are updated together with the base system

Thanks for the quick reply!
That sounds great! I've tried Nix and Guix before and this seems very interesting.