Zero downtime deploys with gunicorn

gunicorn.py

import multiprocessing
import os, os.path
import signal

# The Access log file to write to. [None]
accesslog='{}/logs/unicorn-access.log'.format(os.getcwd())

# The Access log format . [%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"]
# access_log_format='"%(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"'

# Logging config file.
# logconfig="/opt/example/log.conf"

# The Error log file to write to. [-]
errorlog='{}/logs/unicorn-error.log'.format(os.getcwd())

# The granularity of Error log outputs. [info]
loglevel='debug'

# The logger you want to use to log events in gunicorn. [simple]
# logger_class='simple'

# A base to use with setproctitle for process naming. [None]
proc_name='your-app-name'

# Load application code before the worker processes are forked. [False]
preload_app=True

# forked. [False]
daemon=False
# daemon=True

# A filename to use for the PID file. [None]
# pidfile='/var/run/example.pid'
pidfile='{}/pids/unicorn.pid'.format(os.getcwd())

# Switch worker processes to run as this user. [0]
# user="example"

# Switch worker process to run as this group. [0]
# group="example"

# A bit mask for the file mode on files written by Gunicorn. [0]
umask=0002

# The socket to bind. [127.0.0.1:8000]
bind='0.0.0.0:8000'
# bind='unix:{}/sockets/unicorn.sock'.format(os.getcwd())

# The maximum number of pending connections.     [2048]
#  - Amazon Linux default=1024 ($ sysctl net.ipv4.tcp_max_syn_backlog)
backlog=2048

# The number of worker process for handling requests. [1]
workers=multiprocessing.cpu_count() * 2 + 1
# workers=1

# The type of workers to use. [sync]
worker_class='gevent'

# The maximum number of simultaneous clients. [1000]
worker_connections=4098

# The maximum number of requests a worker will process before restarting. [0]
max_requests=4098

# Workers silent for more than this many seconds are killed and restarted. [30]
timeout=120

#The number of seconds to wait for requests on a Keep-Alive connection. [2]
keepalive=2

# Timeout for graceful workers restart.
graceful_timeout=30

def pre_fork(server, worker):
    old_pid_file = '{}/pids/unicorn.pid.oldbin'.format(os.getcwd())

    if os.path.isfile(old_pid_file):
        with open(old_pid_file, 'r') as pid_contents:
            try:
                old_pid = int(pid_contents.read())
                if old_pid != server.pid:
                    os.kill(old_pid, signal.SIGQUIT)
            except Exception as err:
                pass

pre_fork=pre_fork

sv-gunicorn-run.jinja

#!/bin/bash
#
# This file is managed by salt, using the unicorn formula template.
# Editing this file by hand is highly discouraged!
#
exec 2>&1

#
# Since unicorn creates a new pid on restart/reload, it needs a little extra
# love to manage with runit. Instead of managing unicorn directly, we simply
# trap signal calls to the service and redirect them to unicorn directly.
#

RUNIT_PID=$$
APPLICATION_NAME={{ unicorn_application }}
APPLICATION_PATH=/var/www/{{ unicorn_application }}/current
UNICORN_CMD={{ unicorn_command }}
CUR_PID_FILE=/var/www/{{ unicorn_application }}/shared/pids/unicorn.pid
OLD_PID_FILE=$CUR_PID_FILE.oldbin

echo "Runit service restarted (PID: $RUNIT_PID)"

function is_unicorn_alive {
  set +e
  if [ -n $1 ] && kill -0 $1 >/dev/null 2>&1; then
    echo "yes"
  fi
  set -e
}

if [ -e $OLD_PID_FILE ]; then
  OLD_PID=$(cat $OLD_PID_FILE)
  echo "Old master detected (PID: $OLD_PID), waiting for it to quit"
  while [ -n "$(is_unicorn_alive $OLD_PID)" ]; do
    sleep 5
  done
fi

if [ -e $CUR_PID_FILE ]; then
  CUR_PID=$(cat $CUR_PID_FILE)
  if [ -n "$(is_unicorn_alive $CUR_PID)" ]; then
    echo "Detected running Unicorn instance (PID: $CUR_PID)"
    RUNNING=true
  fi
fi

function start {
  unset ACTION
  if [ $RUNNING ]; then
    restart
  else
    echo 'Starting new unicorn instance'

    RUNASUID=$(getent passwd {{ user }} | cut -d: -f3)
    RUNASGROUPS=$(id -G {{ group }} | tr ' ' ':')
    ulimit -n 65535
    export HOME={{ home }}
    export PATH={{ virtualenv_home }}/.virtualenvs/{{ unicorn_application }}/bin:$PATH
    source {{ virtualenv_home }}/.virtualenvs/{{ unicorn_application }}/bin/activate

    cd $APPLICATION_PATH

    exec chpst -u :$RUNASUID:$RUNASGROUPS $UNICORN_CMD -c /var/www/{{ unicorn_application }}/current/config/gunicorn.py wsgi:app
    sleep 3
    CUR_PID=$(cat $CUR_PID_FILE)
  fi
}

function stop {
  unset ACTION
  echo 'Initializing graceful shutdown'
  kill -QUIT $CUR_PID

  while [ -n "$(is_unicorn_alive $CUR_PID)" ]; do
    echo '.'
    sleep 2
  done

  echo 'Unicorn stopped, exiting Runit process'
  kill -9 $RUNIT_PID
}

function restart {
  unset ACTION
  echo "Restart request captured, swapping old master (PID: $CUR_PID) for new master with USR2"
  kill -USR2 $CUR_PID

  sleep 2
  echo 'Restarting Runit service to capture new master PID'
  exit
}

function alarm {
  unset ACTION
  echo 'Unicorn process interrupted'
}

trap 'ACTION=stop' STOP TERM KILL
trap 'ACTION=restart' QUIT USR2 INT
trap 'ACTION=alarm' ALRM

[ $RUNNING ] || ACTION=start

if [ $ACTION ]; then
  echo "Performing \"$ACTION\" action and going into sleep mode until new signal captured"
elif [ $RUNNING ]; then
  echo "Going into sleep mode until new signal captured"
fi

if [ $ACTION ] || [ $RUNNING ]; then
  while true; do
    [ "$ACTION" == 'start' ] && start
    [ "$ACTION" == 'stop' ] && stop
    [ "$ACTION" == 'restart' ] && restart
    [ "$ACTION" == 'alarm' ] && alarm
    sleep 2
  done
fi

Zero downtime deploys with gunicorn.markdown

Zero downtime deploys with gunicorn

Below are the actual files we use in one of our latest applications at Agora Games to achieve zero downtime deploys with gunicorn. I hope these files and notes help. I am happy to update these files or these notes if there are comments/questions. YMMV (of course).

Salient points for each file:

• gunicorn.py: The pre_fork function looks for gunicorn's old PID file in the proper file and sends the proper QUIT signal to the old process once the new process is running.
• sv-gunicorn-run.jinja: This is the runit template we use in our Salt-managed infrastructure for handling the application process management. You could just as easily convert this to a non-templatized version.

I wonder if anyone got this working with a socket. When a USR2 is sent that spins up the oldbin just fine, but when you send the QUIT to it that kills the socket and the socket is never re-stablished with this method.