Objectives

Easily provide and manage free SSL certs with Let's Encrypt for all virtual hosts
Easily independently manage all virtual hosts as separate processes and user accounts -- which enables separate versions and libraries for various stages of the development life-cycle
Easily spin up a new virtual host without opening ports or keeping track of all the port numbers that each app listens on -- all behind a single IP

What's Needed?

nginx-1.15.12+
certbot-auto 0.38.0+
firewall passing 80/443 to nginx
Web Apps

nginx

You might need an nginx that is newer than your distro provides. If so, get it from nginx's repo:

$ sudo apt install curl gnupg2 ca-certificates lsb-release
$ echo "deb http://nginx.org/packages/mainline/ubuntu `lsb_release -cs` nginx"     | sudo tee /etc/apt/sources.list.d/nginx.list
$ curl -fsSL https://nginx.org/keys/nginx_signing.key | sudo apt-key add -
$ sudo apt update
$ sudo apt install nginx

Put this in your /etc/nginx/sites-available and enable it by symlinking it to /etc/nginx/sites-enabled. (You may need to add include /etc/nginx/sites-enabled/*; to /etc/nginx/nginx.conf.)

server {
  # Listen on 80, and 443 with SSL
  listen 80 default_server;
  listen [::]:80 default_server;
  listen 443 default_server ssl;
  listen [::]:443 default_server ssl;

  # Use SNI-based certificates
  ssl_certificate /etc/letsencrypt/live/$ssl_server_name/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/$ssl_server_name/privkey.pem;

  location ~ /.well-known {
    # Serve all ACME requests from this location
    root /var/www/letsencrypt/$http_host;
    allow all;
  }

  location / {
    # Proxy all requests to a unix socket at this location
    proxy_pass http://unix:/var/tmp/nginx/$http_host;
    # Set some special headers for proxying. These are all very standard.
    proxy_http_version 1.1; 
    proxy_set_header Upgrade $http_upgrade; 
    proxy_set_header Connection "upgrade"; 
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

Restart nginx: $ sudo /etc/init.d/nginx restart

nginx.conf explanation

Let's Encrypt puts its certs in specific places under specifically /etc/letsencrypt The ability to use variables in ssl_certificate and ssl_certificate_key directives is new in 1.15.9. Use 1.15.12+ as they fixed some bugs. The $ssl_server_name "returns the server name requested through SNI (1.7.0);"

As is commonly seen in sample reverse proxy configurations, every request is handled by the reverse proxy configuration.

If the User-Agent does not contain letsencrypt, reverse proxy the request to a unix socket where the name of the unix socket is the hostname (e.g. www.example.com is a file of type socket). Of course, a web server will need to be configured to be listening on this socket. These sockets need to be r/w by the user that nginx runs as (e.g. nginx or www-data).

If you don't want to use unix sockets (perhaps the app is on another system): proxy_pass http://$http_host; For this to work, external DNS for your domain will need to point to your reverse proxy server (a.b.c.d), and internal DNS of that same domain name will need to point to your app running internally (w.x.y.z:n, or 127.0.0.1:3000). All virual hosts will need to listen on the same port. Doesn't matter what it is, but they must all be the same port (one reason I like Unix sockets is I don't have to concern myself with that). proxy_pass http://$http_host:3000; You can't proxy_pass to an IP because you can only have one app listening on a given IP/Port.

Web Apps

I use Perl and Mojolicious for all my web apps, but feel free to use whatever. e.g. it's not uncommon to need to host a PHP app like WordPress along-side your other custom apps. Just make your PHP app listen on a socket like described in the nginx section.

Mojolicious is a high-quality, fast, secure, scalable and production-ready web server. It's also great for rapid prototyping development servers. I like to have my production apps be served with Toadfarm (a wrapper on top Mojolicious' hypnotoad) because I can mount one or more Mojolicious apps into a single process. But sometimes I serve my production apps with straight hypnotoad. For development, I use a mix of the morbo web server and the daemon command. With this setup, I can run all of those at the same time, each using their own Perl process so one app can be restarted without affecting the others, and so each app can have it's own Perl binary, version, and libraries. Run a production app on Perl 5.22 with Mojolicious 8.18 and a development version on Perl 5.30 with Mojolicious 8.23. Ideally, run your apps with perlbrew and carton to keep all that straight and simply managed. I'll cover that in this document as well.

The focus of this document is on Let's Encrypt certs and reverse-proxied virtual hosts on multiple processes. For that, I'll use the Mojolicious daemon for all my examples. You can use hypnotoad, toadfarm, morbo, prefork, apache, nginx, or whatever HTTP server you want. Your app can even use WebSockets!

Start-up a Web Server!

Set MOJO_REVERSE_PROXY=1

Tell your application about the presence of a reverse proxy by setting the environment variable MOJO_REVERSE_PROXY. The MOJO_REVERSE_PROXY environment variable can be used to enable proxy support, this allows Mojolicious to automatically pick up the X-Forwarded-For and X-Forwarded-Proto headers.

Also, listen on a unix socket with the hostname as the filename.

$ MOJO_LOG_LEVEL=debug MOJO_REVERSE_PROXY=1 mojo daemon -m development -l http+unix://%2Fvar%2Ftmp%2Fnginx%2Fexample.com
Server available at /var/tmp/nginx/example.com

If your app needs multiple domains or you're running a production Toadfarm with multiple separate apps all mounted into a single Perl process group, just listen on multiple unix sockets, one for each domain that the Perl process group needs to be aware of.

$ MOJO_LOG_LEVEL=debug MOJO_REVERSE_PROXY=1 mojo daemon -m development -l http+unix://%2Fvar%2Ftmp%2Fnginx%2Fexample.com -l http+unix://%2Fvar%2Ftmp%2Fnginx%2Fwww.example.com -l http+unix://%2Fvar%2Ftmp%2Fnginx%2Fapi.example.com -l http+unix://%2Fvar%2Ftmp%2Fnginx%2Ffoobar.com
Server available at /var/tmp/nginx/example.com
Server available at /var/tmp/nginx/www.example.com
Server available at /var/tmp/nginx/api.example.com
Server available at /var/tmp/nginx/foobar.com

The web server will create the named unix socket file in the path provided. Make sure it's owned by the same user running the nginx service.

## For example:
$ sudo chown -R www-data.www-data /var/tmp/nginx

You can start more web servers, at any time, to your heart's content. Just make sure the socket is owned by the nginx service user.

$ MOJO_LOG_LEVEL=debug MOJO_REVERSE_PROXY=1 mojo daemon -m development -l http+unix://%2Fvar%2Ftmp%2Fnginx%2Fww1.example.com &
Server available at /var/tmp/nginx/ww1.example.com
$ MOJO_LOG_LEVEL=debug MOJO_REVERSE_PROXY=1 mojo daemon -m development -l http+unix://%2Fvar%2Ftmp%2Fnginx%2Fww2.example.com &
Server available at /var/tmp/nginx/ww2.example.com
$ sudo chown -R www-data.www-data /var/tmp/nginx
## No need to restart or reload nginx!

Now you can hit your web apps locally:

$ mojo get http+unix://%2Fvar%2Ftmp%2Fnginx%2Fww1.example.com
Your Mojo is working!

Or from a web client outside the infrastructure:

$ mojo get http://ww1.example.com
Your Mojo is working!

In both cases, the web app listening on the ww1.example.com unix socket will handle the request. Including handling WebSockets!

Notice...

If you make the request with a letsencrypt User-Agent, you'll get a 404! Remember, the nginx config file wants to serve static files to letsencrypt User Agents directly from /var/www/letsencrypt/$http_host.

$ mojo get -H "User-Agent: letsencrypt" http://ww1.example.com
...404...

Notice that nginx didn't need to get restarted nor did any additional ports need to get opened. Yes, you do need to make sure that DNS resolves your domain names to this server infrastructure correctly, but you can do that with wildcards and CNAMES to keep it pretty easy and avoid having to be touched for every new web app.

Also notice that if you update some code -- perhaps using morbo to make that simple as well -- only the web app being updated needs to get restarted, not your entire farm.

Also notice that this web app might have a completely different set of Perl libraries -- perhaps a bleeding edge version of Mojolicious, or perhaps one super old and unsupported.

perlbrew and carton

Coming soon...

certbot-auto

For each of the named unix sockets, go fetch a new cert.

$ sudo wget https://dl.eff.org/certbot-auto -O /usr/local/sbin/certbot-auto
$ sudo chmod a+x /usr/local/sbin/certbot-auto
$ sudo mkdir /var/www/letsencrypt
$ sudo chown www-data.www-data /var/www/letsencrypt
$ sudo chown -R www-data.www-data /var/tmp/nginx
$ for host in /var/tmp/nginx/*; do \
    [ -d /var/www/letsencrypt/$host ] && continue; \
    mkdir -p /var/www/letsencrypt/$host; \
    certbot-auto certonly -a webroot --webroot-path=/var/www/letsencrypt/$host -d $host; \
  done
$ sudo chown -R www-data.www-data /etc/letsencrypt
$ sudo /etc/init.d/nginx reload

And renew all the Let's Encrypt certs regularly. For example, drop this in /etc/cron.weekly:

#!/bin/sh
/usr/local/sbin/certbot-auto renew
/etc/init.d/nginx reload

Firewall passing 80/443 to nginx

configuring iptables / ufw