Automatic SSL Certificates for OctoPrint with Step and HAProxy

I wanted to use HTTPS with my OctoPrint setup so I set up a local certificate authority with Step, which you will need for this guide to work. Much of this is based on guides from both HAProxy and Step, combined with some elbow grease on my end to adapt them for my particular setup. Assuming you have both the local certificate authority and a working instance of HAProxy 2.6 or newer (whatever's in Debian stable right now) already set up, read on:

0. SSH into your OctoPrint machine or whatever else is running HAProxy

1. Bootstrap the client:

step ca bootstrap --ca-url https://YOUR_CA_AUTHORITY --fingerprint THE_FINGERPRINT_FROM_THE_CA_AUTHORITY

This will need the fingerprint (and password later) that you received during the CA setup process in the link in the preamble.

2. Generate the certificate:

step ca certificate octopi --san $(hostname -s) --san $(hostname -s).local --san $(hostname -I | cut -d ' ' -f 1) --san $(hostname -I | cut -d ' ' -f 2) --san $(hostname -I | cut -d ' ' -f 3) haproxy.crt haproxy.key --password-file .password

This asssumes the password to your CA authority's private key is cached in a file called .password, see the step ca certificate docs for other ways to generate the key. The filenames for the output (haproxy.crt and haproxy.key) should match the service name for your webserver, in this case haproxy.service. The --san arguments should include any alternate names for your client, which in this case uses hostname, hostname.local, and the 3 IP addresses returned by hostname -I (IPv4, IPv6, IPv6 local). Replace as needed.

3. Move the certificate:

sudo mv haproxy.crt haproxy.key /etc/step/certs

This requires the /etc/step/certs folder to be created, create it if it doesn't exist already.

4. Create the certificate renewal services:

Make sure to replace usernames and hostnames as required:

/etc/systemd/system/[email protected]

[Unit]
Description=Certificate renewer for %I
After=network-online.target
Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production
StartLimitIntervalSec=0
; PartOf=cert-renewer.target

[Service]
Type=oneshot
User=root

Environment=STEPPATH=/etc/step-ca \
            CERT_LOCATION=/etc/step/certs/%i.crt \
            KEY_LOCATION=/etc/step/certs/%i.key \
            ROOT_LOCATION=/home/YOUR_USERNAME/.step/certs/root_ca.crt \
            CA_URL=https://YOUR_CA_AUTHORITY

; ExecCondition checks if the certificate is ready for renewal,
; based on the exit status of the command.
; (In systemd <242, you can use ExecStartPre= here.)
ExecCondition=step certificate needs-renewal ${CERT_LOCATION}

; ExecStart renews the certificate, if ExecStartPre was successful.
ExecStart=step ca renew --force --ca-url ${CA_URL} --root ${ROOT_LOCATION} ${CERT_LOCATION} ${KEY_LOCATION}

; Try to reload or restart the systemd service that relies on this cert-renewer
; If the relying service doesn't exist, forge ahead.
; (In systemd <229, use `reload-or-try-restart` instead of `try-reload-or-restart`)
ExecStartPost=/usr/bin/env sh -c "! systemctl --quiet is-active %i.service || systemctl try-reload-or-restart %i"

[Install]
WantedBy=multi-user.target

/etc/systemd/system/[email protected]

[Unit]
Description=Timer for certificate renewal of %I
Documentation=https://smallstep.com/docs/step-ca/certificate-authority-server-production
; PartOf=cert-renewer.target

[Timer]
Persistent=true

; Run the timer unit every 15 minutes.
OnCalendar=*:1/15

; Always run the timer on time.
AccuracySec=1us

; Add jitter to prevent a "thundering hurd" of simultaneous certificate renewals.
RandomizedDelaySec=5m

[Install]
WantedBy=timers.target

5. Create the HAProxy support scripts:

by default, the above service and timer combination would restart whichever service is attached to it once the certificate is renewed; by using HAProxy's command line API we can instead dynamically load new certificates for HAProxy without restarting. Regardless, HAProxy needs us to concatenate the .crt and .key files into a single .pem, so we add a simple Python script (couldn't get piped output to work as a service) to do so.

/etc/step/certs/merge_certs.py

#!/usr/bin/python3

import os
import sys

def main(certname):
        here = """/etc/step/certs"""
        certname = os.path.join(here, certname)
        crt = certname + ".crt"
        key = certname + ".key"
        pem = certname + ".pem"
        if os.path.exists(pem):
                os.remove(pem)
        with open(crt, "rb") as afile:
                q = afile.read()
        with open(key, "rb") as afile:
                w = afile.read()
        with open(pem, "wb") as afile:
                afile.write(q)
                afile.write(w)

if __name__ == "__main__":
        main(sys.argv[1])

/etc/step/certs/haproxy.sh

#!/bin/bash
python3 /etc/step/certs/merge_certs.py "haproxy"

/etc/step/certs/update-haproxy.sh NOTE: to speak to the HAProxy API you would need to install socat first.

#!/bin/bash

echo -e "set ssl cert /etc/step/certs/haproxy.pem set ssl cert /etc/step/certs/haproxy.pem <<\n$(cat /etc/step/certs/haproxy.pem)\n" | socat stdio /var/run/haproxy/admin.sock
echo -e "commit ssl cert /etc/step/certs/haproxy.pem" | socat stdio /var/run/haproxy/admin.sock

6. Create the certificate renewal service override:

/etc/systemd/system/[email protected]/override.conf NOTE: create /etc/systemd/system/[email protected] folder first.
```
[Service]
;Don't restart the service as usual
ExecStartPost=
;Generate PEM file
ExecStartPost=bash /etc/step/certs/haproxy.sh
;Write new PEM file to HAProxy memory and commit
ExecStartPost=bash /etc/step/certs/update-haproxy.sh
```
With this override, instead of restarting haproxy.service, we instead generate a new PEM file from our freshly renewed certificate and key, and tell HAProxy to load that into memory.

7. Update HAProxy configuration to use SSL:

/etc/haproxy/haproxy.cfg NOTE: this is a truncated configuration file that only shows what to add to frontend and backend, for more check HAProxy docs.
```
frontend public
        ...
        bind :::80 v4v6
        bind :::443 v4v6 ssl crt /etc/step/certs/haproxy.pem
        redirect scheme https if !{ hdr(Host) -i 127.0.0.1 } !{ ssl_fc }
        ...
backend octoprint
        ...
        acl needs_scheme req.hdr_cnt(X-Scheme) eq 0
        http-request replace-path /octoprint/(.*) /\1
        http-request set-header X-Script-Name /octoprint
        http-request add-header X-Scheme https if needs_scheme { ssl_fc }
        http-request add-header X-Scheme http if needs_scheme !{ ssl_fc }
        server octoprint1 127.0.0.1:5000
        ...
```
For the frontend, the above tells HAProxy to use the .pem file we generated for the certificate when accessing the client on port 443, except for if the client itself is accessing via loopback. For the backend, the http-request lines are there to get OctoPrint's scripting working with the certificates, consult the reverse proxy FAQs for more. If you found a guide somewhere that uses reqadd instead of http-request, that guide is using a super old version of HAProxy, so be warned.

8. Create the .pem file:

cd /etc/step/certs
sudo bash haproxy.sh

9. Enable the services:

sudo systemctl daemon-reload
sudo systemctl enable --now [email protected]

If everything went well you should get short-lived TLS certificates that not only automagically renew in the background whenever they're approaching expiry, but also notify HAProxy that they've renewed and to use the new certificates as soon as they do so.

Depending on which plugins you have installed you may need to make some configuration changes; as an example Pretty GCode breaks with this setup until you edit the plugin's hardcoded paths.

Bonus: Securing mjpg_streamer

10. Install stunnel:

sudo apt install stunnel

11. Setup stunnel config:

/etc/stunnel/webcamd.conf

key = /etc/step/certs/haproxy.pem
cert = /etc/step/certs/haproxy.pem
CAfile = /root/.step/certs/root_ca.crt
debug = 7
sslVersion = all
; needed to make this a service
foreground = yes

[webcamd]
client = no
; port that you will access using web UI
accept = 8975
; port that mjpg_streamer is streaming to
connect = 8080

12. Setup stunnel service:

sudo systemctl enable --now stunnel@webcamd

13. Restart stunnel upon updating certificate:

/etc/systemd/system/[email protected]/override.conf

[Service]
;Don't restart the service as usual
ExecStartPost=
;Generate PEM file
ExecStartPost=bash /etc/step/certs/haproxy.sh
;Write new PEM file to HAProxy memory and commit
ExecStartPost=bash /etc/step/certs/update-haproxy.sh
;Reload STunnel services, which use the same cert
ExecStartPost=service stunnel@webcamd restart

14. Add root certificate to cert bundles:

Without this step Octoprint will throw up SSL verify errors trying to access timelapse snapshots.

If using the system CA certificates bundle:

sudo cp ~/.step/certs/root_ca.crt /usr/local/share/ca-certificates
sudo update-ca-certificates

If using certifi from within OctoPrint's venv:
```
source ~/OctoPrint/venv/bin/activate
cat ~/.step/certs/root_ca.crt | tee -a $(python -c "import requests;print(requests.certs.where())")
```
Note, this step would need to be re-done anytime there's a change to certifi within the OctoPrint virtualenv.

15. Configure Octoprint

Note the accept = PORT line from the stunnel configuration, you should tell your webcam plugin and anything else expecting http://hostname.local:8080/?action=stream to use https://hostname.local:8975/?action=stream.

thurask/README.md