Systemd for github deployments

systemd.md

Using systemd + journald for GitHub deployments

At https://svsticky.nl/ we heavily make use of Systemd for all kinds of sysadmin tasks. We deploy the website from the https://github.com/svsticky/static-sticky repository, and people working on the website want to know why deployments failed if they do.

On https://github.com/svsticky/static-sticky/deployments you can find a list of deployments that were created for the website. If you click on the deployment, you can find the logs of the deployment. E.g. https://svsticky.nl/logs/acd2a99e348c40c58d6b13e4bd896a7c

How did we achieve this using systemd?

Deployment script as a oneshot unit

First, we create a oneshot service, that executes the deploy script. How the actualy deploy script should look like, we will now not dive into, as it is not the meat of what I want to show today.

# deploy-website.service
[Unit]
Description=Deployment of static sticky
[Service]
ExecStart=/usr/local/bin/deploy-static-sticky.sh

This script can then be triggered by either creating a corresponding deploy-website.path unit that listens to changes of a repository, or a deploy-website.timer unit that deploys the website periodically. Either works.

A unique URL to reach the logs

Every systemd unit run has an associated InvocationID that uniquely identifies a run of that unit. Journald allows us to request the logs of a specific invocation id.

journalctl _SYSTEMD_INVOCATION_ID=$INVOCATION_ID

Systemd will set the $INVOCATION_ID enviroment variable for the program that is in ExecStart. Hence our deploy script has access to its own invocation ID, and can tell GitHub about it ( Full code https://github.com/svsticky/sadserver/blob/master/ansible/templates/usr/local/bin/deploy-static-sticky.sh.j2) :

# deploy-static-sticky.sh
# Full code : 
curl \
  --silent --show-error \
  --output /dev/null \
  --fail \
  --user "${GITHUB_USER}:${GITHUB_PASSWORD}" \
  --header "Accept: application/vnd.github.ant-man-preview+json" \
  --header "Content-Type: application/json" \
  --request POST \
  --data \
    '{"state": "success",
      "environment_url": "https://{{ canonical_hostname }}/",
      "log_url": "https://{{ canonical_hostname }}/logs/'"${INVOCATION_ID}"'"
    }' \
"https://api.github.com/repos/svsticky/static-sticky/deployments/${DEPLOY_ID}/statuses"

Serving the logs using systemd-journal-gatewayd and nginx

Systemd comes with a unit systemd-journal-gatewayd.socket that exposes the Journal API as an HTTP service. Of course, we do not want to expose the entire journal API to the internet, for security reasons. However, we want people to be able to request the logs of a unit run, if they know the (unique) INVOCATION_ID of a unit. So we have the following rule in nginx, that requests the logs from systemd-journal-gatewayd and nothing else.

  location /logs/ {
    # Construct correct request URI for systemd-journald-gateway, while only
    # allowing alphanumerics (to prevent exposing the other endpoints and
    # parameters that systemd-journald-gateway supports
    rewrite "^/logs/([\d\w]{32})$" /entries?_SYSTEMD_INVOCATION_ID=$1 break;
    return 403;
    proxy_pass http://localhost:19531;
  }

Conclusion

We have automatic deployments for https://svsticky.nl. We have a little script that does this deployment that is run as a systemd oneshot service. This oneshot unit can be triggered by .path or .timer units or manually triggered. The oneshot service has access to $INVOCATION_ID which uniquly identifies the log entries for this run in journald. journald can expose logs as an http web service through systemd-journal-gatewayd.socket and systemd-journal-gatewayd.service. We then use nginx as a proxy in front of this to make sure that people can only view logs of a deployment of which they know the $INVOCATION_ID.

This setup touches many of the strengths of systemd. We touch all kinds of unit types (.path, .timer , .socket and .service), and make use of its structured logging facilities. systemd does all the heavy lifting for us, and with a small nginx rule, we now can link logs of deployments directly to the Github UI.

Come to think of it, the nginx rule should probably hardcode the _SYSTEMD_UNIT name as well when you use this for one unit, just to be sure :)