Deploying Web Services on Docker with nginx-proxy

Note: This is an effort on keeping better notes on things I've set up and what works; please don't expect anything in this doc to be "right" or best practice. It's better to keep this than dig back through zsh history. :)

Project

I recently deployed mkwords, a web application built fully in Clojure/ClojureScript for selecting random words from a high quality default wordlist; it's built around hazard, the Clojure version of my old node-xkcd-password library. Seemed fitting for my first Clojure lib of any real substance to mirror my first node lib.

Additionally, I wanted a reason to try out Let's Encrypt since they were giving out beta invites. To throw another wrench in, I opted to deploy with docker, another first.

Building mkwords

mkwords was built using the default Reagent template; it was an easy place to start since the leiningen project was basically all set up for me, to scaffold it, I just ran:

lein new reagent mkpass

Originally, the project was called mkpass rather than mkwords; when I decided to change this later it was a very easy project-wide search/replace; no other changes necessary.

The scaffold came with a few simple out-of-the-box views so you could get a feel for how a Reagent project was set up. I've had no experience with Facebook's React (of which Reagent is a ClojureScript wrapper), but we use Ractive at Urban Airship, and many of the concepts we use there are analogous to Facebook's Flux architecture. I felt very at home with Reagent almost immediately (and its use of atoms for its state).

Getting a development server up and running was easy; in two separate terminals I ran lein run and lein figwheel which lifted the live-rebuilding ring and frontend build servers, respectively.

I found the backend auto-rebuilding to be more than adequate; it never got itself into an undefined state throughout the whole project. I only had to stop it when I added a new dependency to the leiningen project. The frontend server was another story entirely:

When the rebuilding worked, I still had to do a hard-refresh of the page to get it back into a usable state. The auto-reload would function, but for some reason it would fail to re-execute the initial XHR that retrieves the wordlist, getting things stuck in a non-working state.
When updating deps, restarting the build server wasn't enough. I needed to run a (reset-autobuild) from within the REPL for changes to get picked up; I assume it was running an "only rebuild what you need" that wasn't catching these dep changes, even after a restart of the process.
Many other times where I got into undefined states, necessitating more (reset-autobuild) steps.

All in all though: pretty smooth, especially for a Clojure beginner.

Introducing Node.js

In the end, I broke from a fully-Clojure setup. For reasons detailed later, I was unable to use the out-of-the-box minified version of bijou—the very-lightweight responsive framework I chose—so I needed to build SCSS. All of the options I found for doing this from Clojure were not well maintained and needed other dependencies (either Ruby or JRuby), so in the end I added a package.json and just used node-sass, which is highly reliable and bundles libsass for compilation. This does require a build step if there isn't a redistributable for your environment, but their is for most you'll run into.

I ended up including this step by using lein-shell, a Leiningen plugin which can run shell commands as part of build steps. This worked immediately and perfectly, and I was on my way. I did remove a number of things from my Leiningen project that weren't necessary anymore because of this (mostly lein-asset-minifier).

Building for distribution

The Leiningen project I was using was already set to bundle an "uberjar", which is just a jar with all dependencies bundled up and ready to be put into production. This process is really painless. Just do an:

lein uberjar

…and everything is built up into a single redistributable, including all of your static assets. I'm really impressed by how easy that process was.

Once bundled, the jar sat at target/mkwords.jar; I could do a test-run of this with:

java -cp target/mkwords.jar clojure.main -m mkwords.server

This spun up my complete service on http://localhost:3000, ready to test before going into production.

Building the Docker Image

There were several workflows that I found, but the one that I tried out came directly from the official clojure docker image. They detail a few different ways on that page to run your app on the image with Leiningen, or to build it directly on the image.

In the end I opted for something even more simple: building on my local machine and then just copying the jar to the Docker image. This wouldn't be a good idea for some sort of CI, but for my purposes it worked great.

Since we're just deploying an uberjar, I ended up changing to just the official java:8 docker container; clojure tools aren't necessary if I'm building locally.

You can see the build script and the Dockerfile, they're very boring. More on the Dockerfile later though.

Deploying with Docker

Once the dockerfile is built, you can run it locally (assuming docker is installed) with a:

docker run --name=mkwords -p 3000:3000 -i -t fardog/mkwords

To work with the docker image, there's some commands worth knowing:

docker ps -a  # show all docker containers, running or not
docker stop <container_name_or_hash>  # stop a running container
docker images  # show available images
docker rm <container_name_or_hash>  # remove a container
docker rmi <image_name_or_hash>  # remove an image

Satisfied with that, I decided to move to actual deployment; I have a Digital Ocean account (which runs my twitterbot primes and some other services); I opted to spin up a new box using their "Ubuntu Docker 1.9.0 on 14.04" image; as the name would imply it already has Docker on the box.

From my local machine, I pushed up my newly created Docker image (after creating the repo through the Docker Hub UI):

docker login  # enter your credentials
docker push fardog/mkwords

On the Digital Ocean box, I then did the following:

docker pull fardog/mkwords
docker run -p 3000:3000 --name=mkwords fardog/mkwords

Huge success: it was available on my remote server at http://mkwords.fardog.io:3000

Now obviously I didn't want to expose the Jetty server to the world just like that: first off, it should be behind a reliable webserver like nginx, and secondly it should be behind SSL.

Generating Certificates with Let's Encrypt

Let's Encrypt is still in beta, so I'm censoring a few things in these; but wow: it is dead simple. I'm really impressed with their work here:

git clone https://github.com/letsencrypt/letsencrypt  # clone the repo
cd letsencrypt/
./letsencrypt-auto --server <directory_server_url> --help  # showed some help
# now lets genrate the certificate
./letsencrypt-auto certonly -a standalone -d mkwords.fardog.io --server <directory_server_url>

That was it; it spun up a webserver automatically to prove I was at the domain I said I was (DNS had to be pointing here first obviously) and then generated the certificates. Done and amazingly done.

Running a Dockerized nginx proxy

At this point: I just really wanted to see the thing work! I decided on the nginx-proxy docker image, because it did a lot of out-of-the-box magic to get things up and running without requiring additional configuration; I plan to revisit this someday to better understand how it actually works, but for now I was able to get running very quickly.

First off, letsencrypt genrates all of its certificates with a .pem extension; this is fine: they're actually already in the format you need, they just need to be renamed.

The nginx-proxy image matches things up by having names passed around that match the domains it'll be serving; you'll see the string mkwords.fardog.io all over in the commands setting it up.

So I copied the certificate and private key to their reseing place on the filesystem:

cp fullchain.pem /etc/web-certs/mkwords.fardog.io.crt
cp privkey.pem /etc/web-certs/mkwords.fardog.io.key

Once that was done, I ran the nginx-proxy docker image, passing those certificate paths:

docker pull jwilder/nginx-proxy
docker run -d -p 80:80 -p 443:443 -v /etc/web-certs:/etc/nginx/certs \
  -v /var/run/docker.sock:/tmp/docker.sock:ro --name nginx-proxy jwilder/nginx-proxy

That got up and running. Then I ran my mkwords docker image, passing the correct configuration parameters to identify it:

docker run -e VIRTUAL_HOST=mkwords.fardog.io --name mkwords fardog/mkwords

That was it; I visited https://mkwords.fardog.io and there it was!

n.b. There's a notable thing in how all this works: when running the mkwords container, I'm not exposing any of the ports via the CLI; if you check the Dockerfile you'll see an EXPOSE directive; the port that's exposed here is exposed over docker's private internal network. Its with this port that the nginx proxy is able to expose your service, and this service won't be exposed to the world except through nginx.

one problem…

Whoops, broke my fonts in Chrome. Turns out that the SCSS framework I chose was loading fonts from Google over HTTP, not HTTPS. This is what drove me to build SCSS rather than using the already-created minified version.

Surviving a Restart

Now that I had everything running, I wanted to ensure that things could be started more easily. Given that I've named my two docker containers with sensible names, it's straightforward to create upstart scripts to run them:

# file /etc/init/nginx-proxy.conf
description "nginx proxy"
author "Nathan Wittstock"
start on filesystem and started docker
stop on runlevel [!2345]
respawn
script
  /usr/bin/docker start -a nginx-proxy
end script

# file /etc/init/mkwords.conf
description "mkwords"
author "Nathan Wittstock"
start on filesystem and started docker and started nginx-proxy
stop on runlevel [!2345]
respawn
script
  /usr/bin/docker start -a mkwords
end script

Now you could (ideally) do the following (assuming your images weren't running already):

start nginx-proxy
start mkwords
stop mkwords
stop nginx-proxy

In my case starting works, stopping doesn't. I still need to do a docker stop <container_name> to stop things; I haven't had the chance to look into this yet.

Updating your Application Container

I haven't figured out how to do this in a way that seems clean yet. My current process has been:

Run the build script in my repo and push it to docker hub:

./build-docker-image.sh
docker push fardog/mkwords

Then update the image on the server by stopping/pulling/starting:

docker pull fardog/mkwords  # pull the updated image
stop mkwords  # stop the upstart script
docker stop mkwords  # since my upstart script doesn't kill it yet :/
docker rm mkwords  # remove the current container
# then start a new container which will use the new image
docker run -e VIRTUAL_HOST=mkwords.fardog.io --name mkwords fardog/mkwords

Emphatic "bleh". There has to be a more elegant way to do that.

Conclusion: It works! There could be improvements to the process. In total though, I'm becoming much more familiar with Clojure. I'm feeling a lot of parallels with when I started learning Node.js several years ago: lots of getting things done without knowing if you're doing it even remotely right. Learning!

fardog/docker-nginx-proxy.md