NGINX+SPDY with Unicorn. True Zero-Downtime unless migrations. Best practices.

0. nginx_setup.sh

# Nginx+Unicorn best-practices congifuration guide. Heartbleed fixed.
# We use latest stable nginx with fresh **openssl**, **zlib** and **pcre** dependencies.
# Some extra handy modules to use: --with-http_stub_status_module --with-http_gzip_static_module
#
# Deployment structure
#
# SERVER:
#  /etc/init.d/nginx                             (1. nginx)
#  /home/app/public_html/app_production/current  (Capistrano directory)
#
# APP:
#  config/server/production/nginx.conf           (2. nginx.conf)
#  config/server/production/nginx_host.conf      (3. nginx_host.conf)
#  config/server/production/nginx_errors.conf    (4. nginx_errors.conf)
#  config/deploy.rb                              (5. deploy.rb)
#  config/deploy/production.rb                   (6. production.rb)
#  config/server/production/unicorn.rb           (7. unicorn.rb)
#  config/server/production/unicorn_init.sh      (8. unicorn_init.sh)






cd /usr/src
wget http://nginx.org/download/nginx-1.5.13.tar.gz
tar xzvf ./nginx-1.5.13.tar.gz && rm -f ./nginx-1.5.13.tar.gz

wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.32.tar.gz
tar xzvf pcre-8.32.tar.gz && rm -f ./pcre-8.32.tar.gz

wget http://www.openssl.org/source/openssl-1.0.1g.tar.gz
tar xzvf openssl-1.0.1g.tar.gz && rm -f openssl-1.0.1g.tar.gz

cd nginx-1.5.13
./configure --prefix=/opt/nginx --with-pcre=/usr/src/pcre-8.32 --with-openssl-opt=no-krb5 --with-openssl=/usr/src/openssl-1.0.1g --with-http_ssl_module --with-http_spdy_module --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --with-http_stub_status_module --with-http_gzip_static_module

make && make install

mkdir /tmp/client_body_temp
mkdir /opt/nginx/ssl_certs 

echo "include /home/app/public_html/app_production/current/config/server/production/nginx.conf;" > /opt/nginx/conf/nginx.conf 

vim /etc/init.d/nginx # see the config below
chmod +x /etc/init.d/nginx && update-rc.d -f nginx defaults

1. nginx

#! /bin/sh

### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the nginx web server
# Description:       starts nginx using start-stop-daemon
### END INIT INFO
#
# /etc/init.d/nginx


PATH=/opt/nginx/sbin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/opt/nginx/sbin/nginx
NAME=nginx
DESC=nginx

test -x $DAEMON || exit 0

# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
        . /etc/default/nginx
fi

set -e

case "$1" in
  start)
        echo -n "Starting $DESC: "
        start-stop-daemon --start --quiet --pidfile /opt/nginx/logs/$NAME.pid \
                --exec $DAEMON -- $DAEMON_OPTS
        echo "$NAME."
        ;;
  stop)
        echo -n "Stopping $DESC: "
        start-stop-daemon --stop --quiet --pidfile /opt/nginx/logs/$NAME.pid \
                --exec $DAEMON
        echo "$NAME."
        ;;
  restart|force-reload)
        echo -n "Restarting $DESC: "
        start-stop-daemon --stop --quiet --pidfile \
                /opt/nginx/logs/$NAME.pid --exec $DAEMON
        sleep 1
        start-stop-daemon --start --quiet --pidfile \
                /opt/nginx/logs/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS
        echo "$NAME."
        ;;
  reload)
          echo -n "Reloading $DESC configuration: "
          start-stop-daemon --stop --signal HUP --quiet --pidfile     /opt/nginx/logs/$NAME.pid \
              --exec $DAEMON
          echo "$NAME."
          ;;
      *)
            N=/etc/init.d/$NAME
            echo "Usage: $N {start|stop|restart|reload|force-reload}" >&2
            exit 1
            ;;
    esac

    exit 0

2. nginx.conf

# Nginx main block configuration file.
# The most important directives here are ssl_protocols and ssl_ciphers
# Keep nginx configuration in repo, then just include it in /opt/nginx/conf/nginx.conf
# And DDoS prevent attack with directive limit_req_zone (limit 10 request/sec from 1 IP address)
# then it enables in the server block by "limit_req zone=one".
#
# config/server/production/nginx.conf


user                          app;
worker_processes              2;
worker_priority               -5;
timer_resolution              100ms;

error_log                     logs/nginx.error.log;

events {
    use                       epoll;
    worker_connections        2048;
}


http {
    client_max_body_size      25m;
    client_body_buffer_size   128k;
    client_body_temp_path     /tmp/client_body_temp;

    include                   mime.types;
    default_type              application/octet-stream;
    server_tokens             off;
    sendfile                  on;
    tcp_nopush                on;
    tcp_nodelay               on;
    keepalive_timeout         70;

    gzip                      on;
    gzip_http_version         1.1;
    gzip_disable              "msie6";
    gzip_vary                 on;
    gzip_min_length           1100;
    gzip_buffers              64 8k;
    gzip_comp_level           3;
    gzip_proxied              any;
    gzip_types                text/plain text/css application/x-javascript text/xml application/xml;

    ssl_certificate           /opt/nginx/ssl_certs/server.crt;
    ssl_certificate_key       /opt/nginx/ssl_certs/server.key;
    ssl_session_timeout       15m;
    ssl_protocols             SSLv3 TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_session_cache         shared:SSL:10m;
    ssl_stapling              on;

    add_header                Strict-Transport-Security "max-age=16070400; includeSubdomains";
    add_header                X-Frame-Options DENY;

    limit_req_zone            $binary_remote_addr zone=one:10m rate=50r/s;

    include                   /home/app/public_html/app_production/current/config/server/nginx_host.conf;
}

3. nginx_host.conf

# Nginx server block configuration with proxy_pass to Unicorn upstream
# We use full-SSL site with web-server redirection, no mess with Rails application redirection
#
# config/server/production/nginx_host.conf


upstream unicorn {
  server              unix:/tmp/unicorn.production.sock fail_timeout=0;
}


server {
  listen              80;
  server_name         server.com;
  rewrite             ^(.*) https://$host$1 permanent;

  location ~ \.(php|html)$ {
    deny              all;
  }

  access_log          /dev/null;
  error_log           /dev/null;
}


server {
  ssl                 on;
  listen              443 spdy ssl;
  server_name         server.com;
  root                /home/app/public_html/app_production/current/public;
  try_files           $uri /system/maintenance.html @unicorn;

  location @unicorn {
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_set_header  Host $http_host;
    proxy_redirect    off;
    proxy_pass        http://unicorn;
    limit_req         zone=one burst=15;

    access_log        /dev/null;
    error_log         logs/unicorn.error.log;
  }

  location ~ ^/(assets|images|javascripts|stylesheets|swfs|system)/ {
    gzip_static       on;
    expires           max;
    add_header        Cache-Control public;
    add_header        Last-Modified "";
    add_header        ETag "";
  }

  include             /home/app/public_html/app_production/current/config/server/production/nginx_errors.conf;

  access_log          /dev/null;
  error_log           /dev/null;
}

4. nginx_errors.conf

# nginx configuration piece to handle errorrs
#
# config/server/production/nginx_errors.conf

error_page          500 502 504  /500.html;
error_page          503 @503;

location = /50x.html {
    root            html;
}

location = /404.html {
    root            html;
}

location @503 {
  error_page 405 = /system/maintenance.html;
  if (-f $document_root/system/maintenance.html) {
    rewrite         ^(.*)$ /system/maintenance.html break;
  }
  rewrite           ^(.*)$ /503.html break;
}

if ($request_method !~ ^(GET|HEAD|PUT|POST|DELETE|OPTIONS)$ ){
  return            405;
}

if (-f $document_root/system/maintenance.html) {
  return            503;
}

location ~ \.(php|html)$ {
  return            405;
}

5. deploy.rb

# Capistrano configuration. Now with TRUE zero-downtime unless DB migration.
# 
# require 'new_relic/recipes'         - Newrelic notification about deployment
# require 'capistrano/ext/multistage' - We use 2 deployment environment: staging and production.
# set :deploy_via, :remote_cache      - fetch only latest changes during deployment
# set :normalize_asset_timestamps     - no need to touch (date modification) every assets
# "deploy:web:disable"                - traditional maintenance page (during DB migrations deployment)
# task :restart                       - Unicorn with preload_app should be reloaded by USR2+QUIT signals, not HUP
#
# http://unicorn.bogomips.org/SIGNALS.html
# "If “preload_app” is true, then application code changes will have no effect; 
# USR2 + QUIT (see below) must be used to load newer code in this case"
# 
# config/deploy.rb


require 'bundler/capistrano'
require 'capistrano/ext/multistage'
require 'new_relic/recipes'

set :stages,                     %w(staging production)
set :default_stage,              "staging"

set :scm,                        :git
set :repository,                 "..."
set :deploy_via,                 :remote_cache
default_run_options[:pty]        = true

set :application,                "app"
set :use_sudo,                   false
set :user,                       "app"
set :normalize_asset_timestamps, false


before "deploy",                 "deploy:delayed_job:stop"
before "deploy:migrations",      "deploy:delayed_job:stop"

after  "deploy:update_code",     "deploy:symlink_shared"
before "deploy:migrate",         "deploy:web:disable", "deploy:db:backup"

after  "deploy",                                      "newrelic:notice_deployment", "deploy:cleanup", "deploy:delayed_job:restart"
after  "deploy:migrations",      "deploy:web:enable", "newrelic:notice_deployment", "deploy:cleanup", "deploy:delayed_job:restart"


namespace :deploy do

  %w[start stop].each do |command|
    desc "#{command} unicorn server"
    task command, :roles => :app, :except => { :no_release => true } do
      run "#{current_path}/config/server/#{rails_env}/unicorn_init.sh #{command}"
    end
  end

  desc "restart unicorn server"
  task :restart, :roles => :app, :except => { :no_release => true } do
    run "#{current_path}/config/server/#{rails_env}/unicorn_init.sh upgrade"
  end


  desc "Link in the production database.yml and assets"
  task :symlink_shared do
    run "ln -nfs #{deploy_to}/shared/config/database.yml #{release_path}/config/database.yml"
  end

  namespace :delayed_job do 
    desc "Restart the delayed_job process"
    task :restart, :roles => :app, :except => { :no_release => true } do
      run "cd #{current_path}; RAILS_ENV=#{rails_env} bundle exec script/delayed_job restart" rescue nil
    end
    desc "Stop the delayed_job process"
    task :stop, :roles => :app, :except => { :no_release => true } do
      run "cd #{current_path}; RAILS_ENV=#{rails_env} bundle exec script/delayed_job stop" rescue nil
    end
  end


  namespace :db do
    desc "backup of database before migrations are invoked"
    task :backup, :roles => :db, :only => { :primary => true } do
      filename = "#{deploy_to}/shared/db_backup/#{stage}_db.#{Time.now.utc.strftime("%Y-%m-%d_%I:%M")}_before_deploy.gz"
      text = capture "cat #{deploy_to}/current/config/database.yml"
      yaml = YAML::load(text)["#{stage}"]
    
      on_rollback { run "rm #{filename}" }
      run "mysqldump --single-transaction --quick -u#{yaml['username']} -h#{yaml['host']} -p#{yaml['password']} #{yaml['database']} | gzip -c > #{filename}"
    end
  end


  namespace :web do
    desc "Maintenance start"
    task :disable, :roles => :web do
      on_rollback { run "rm #{shared_path}/system/maintenance.html" }
      page = File.read("public/503.html")
      put page, "#{shared_path}/system/maintenance.html", :mode => 0644
    end
    
    desc "Maintenance stop"
    task :enable, :roles => :web do
      run "rm #{shared_path}/system/maintenance.html"
    end
  end

end


namespace :log do
  desc "A pinch of tail"
  task :tailf, :roles => :app do
    run "tail -n 10000 -f #{shared_path}/log/#{rails_env}.log" do |channel, stream, data|
      puts "#{data}"
      break if stream == :err
    end
  end
end

6. production.rb

# capistrano production config
#
# config/deploy/production.rb


server "8.8.8.8",                :app, :web, :db, :primary => true
set :branch,                     "production"
set :deploy_to,                  "/home/app/public_html/app_production"
set :rails_env,                  "production"

7. unicorn.rb

# Unicorn configuration file to be running by unicorn_init.sh with capistrano task
# read an example configuration before: http://unicorn.bogomips.org/examples/unicorn.conf.rb
# 
# working_directory, pid, paths - internal Unicorn variables must to setup
# worker_process 4              - is good enough for serve small production application
# timeout 30                    - time limit when unresponded workers to restart
# preload_app true              - the most interesting option that confuse a lot of us,
#                                 just setup is as true always, it means extra work on 
#                                 deployment scripts to make it correctly
# BUNDLE_GEMFILE                - make Gemfile accessible with new master
# before_fork, after_fork       - reconnect to all dependent services: DB, Redis, Sphinx etc.
#                                 deal with old_pid only if CPU or RAM are limited enough
#
# config/server/production/unicorn.rb


app_path          = "/home/app/public_html/app_production/current"

working_directory "#{app_path}"
pid               "#{app_path}/tmp/pids/unicorn.pid"
stderr_path       "#{app_path}/log/unicorn.log"
stdout_path       "#{app_path}/log/unicorn.log"

listen            "/tmp/unicorn.production.sock"
worker_processes  4
timeout           30
preload_app       true


before_exec do |server|
  ENV["BUNDLE_GEMFILE"] = "#{app_path}/Gemfile"
end


before_fork do |server, worker|
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
  end

  if defined?(Resque)
    Resque.redis.quit
  end

  sleep 1
end


after_fork do |server, worker|
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.establish_connection
  end
  
  if defined?(Resque)
    Resque.redis           = 'localhost:6379'
  end
end

8. unicorn_init.sh

# Unicorn handle shell script
# 
# APP_ROOT, PID             - are the same as you setup above
# CMD                       - use bundle binstubs (bundle install --binstubs) to 
#                             forget about "bundle exec" stuff, run in demonize mode
#                             bin/unicorn is for Rack application (config.ru in root dir), but 
#                             bin/unicorn_rails is to use with Rails 2.3
#
# To handle "app_preload true" configuration we should use USR2+QUIT signals, not HUP!
# So we rewrite capistrano deployment scripts to manage it.
#
# config/server/production/unicorn_init.sh


#!/bin/sh
set -e
# Example init script, this can be used with nginx, too,
# since nginx and unicorn accept the same signals

TIMEOUT=${TIMEOUT-60}
APP_ROOT=/home/app/public_html/app_production/current
PID=$APP_ROOT/tmp/pids/unicorn.pid
CMD="$APP_ROOT/bin/unicorn -D -c $APP_ROOT/config/server/unicorn.rb -E production"
action="$1"
set -u

old_pid="$PID.oldbin"

cd $APP_ROOT || exit 1

sig () {
        test -s "$PID" && kill -$1 `cat $PID`
}

oldsig () {
        test -s $old_pid && kill -$1 `cat $old_pid`
}

case $action in
start)
        sig 0 && echo >&2 "Already running" && exit 0
        $CMD
        ;;
stop)
        sig QUIT && exit 0
        echo >&2 "Not running"
        ;;
force-stop)
        sig TERM && exit 0
        echo >&2 "Not running"
        ;;
restart|reload)
        sig HUP && echo reloaded OK && exit 0
        echo >&2 "Couldn't reload, starting '$CMD' instead"
        $CMD
        ;;
upgrade)
        if sig USR2 && sleep 15 && sig 0 && oldsig QUIT
        then
                n=$TIMEOUT
                while test -s $old_pid && test $n -ge 0
                do
                        printf '.' && sleep 1 && n=$(( $n - 1 ))
                done
                echo

                if test $n -lt 0 && test -s $old_pid
                then
                        echo >&2 "$old_pid still exists after $TIMEOUT seconds"
                        exit 1
                fi
                exit 0
        fi
        echo >&2 "Couldn't upgrade, starting '$CMD' instead"
        $CMD
        ;;
reopen-logs)
        sig USR1
        ;;
*)
        echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
        exit 1
        ;;
esac

отличный конфиг, букмаркаю, nginx + unicorn, шустрая тема

UPDATE Nginx caching open file descriptors (static assets only), their sizes and modification times with using of open_file_cache

UPDATE unicorn USR2 with followed by sleep for 15sec of 2. This time is enough to preload the application.

при выполнении upgrade поулчаю следующее:

Couldn't upgrade, starting '/home/vagrant/app/current/bin/unicorn_rails -D -c /home/vagrant/app/shared/config/unicorn.rb -E production' instead
master failed to start, check stderr log for details

В логах следующее:

I, [2012-09-12T15:31:02.127414 #1417]  INFO -- : executing ["/home/vagrant/app/current/bin/unicorn_rails", "-D", "-c", "/home/vagrant/app/shared/config/unicorn.rb", "-E", "production", {5=>#<Kgio::UNIXServer:/home/vagrant/app/shared/tmp/sockets/unicorn.sock>}] (in /home/vagrant/app/releases/1)
/usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- bundler/setup (LoadError)
from /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /home/vagrant/app/current/bin/unicorn_rails:14:in `<main>'

В чем может быть проблема?

@alfuken gem install bundler и проверь, что свежий Gemfile.lock в репо

UPDATE: now with SPDY support

@mikhailov, any reason for running Unicorn processes as root? Thanks for sharing.

@xymbol master process runs as root, but workers as app

The init.d scripts start nginx and unicorn but cannot perform the other actions (e.g. stop). Not sure if this is Ubuntu specific or not, but start-stop-daemon will not create a pid file without the flag --make-pid - and even then will write the wrong pid due to the nginx/unicorn processes forking from the main process.

Здравствуйте. Сделал как написано в вашем конфиге, но у меня не стартуют рельсы.
В логах nginx все норм, а вот в логах unicorn.error.log данное сообщение

2013/03/26 15:49:41 [error] 25371#0: *8 limiting requests, excess: 1.000 by zone "one", client: 192.168.91.185, server: localhost, request: "GET / HTTP/1.1", host: "192.168.91.15"

А сама страница отдает 404 Ошибку

Very good. Please change the comment: (limit 10 request/sec from 1 IP address) to (limit 50...) about nginx limit_req_zone.

@k3NGuru по-моему, limit_req_zone был поломан и его починили только несколько месяцев назад, пробуйте стартануть nginx без него

@mikhailov Все равно не заводится система. Уже сделал как написано тут https://coderwall.com/p/8igwqa настроил как написано в данном конфиге, Вот моя папка https://github.com/k3NGuru/mlvz

2013/04/05 16:02:07 [error] 19467#0: *2 limiting requests, excess: 1.000 by zone "one", client: 192.168.91.163, server: 192.168.91.15, request: "GET / HTTP/1.1^@^Dho^@^@", host: "192.168.91.15"

192.168.91.15 - Сервер
192.168.91.163 - Клиент

SSL настроил как написано тут http://wiki.nginx.org/HttpSslModule

@k3NGuru все дело в специфичной работе встроенного модуля limit_req, добавь burst=5 для решения проблемы:

limit_req         zone=one burst=5;

Это пропустит 6 запросов в секунду, надо больше с одной айпи - увеличивай burst, либо распараллель нагрузку между несколькими Nginx серверами. Почему так, а не иначе, попытались объяснить здесь: http://www.lexa.ru/nginx-ru/msg38569.html Сегодня задан вопрос в nginx-рассылку.

Thank you for sharing this. :-)

Amazing. Thank you very much man =)

Great stuff to ease nginx usage!

A few queries/points:

1. I needed to add ' application/javascript' to gzip_types, else js was not getting compressed. Added that before application/x-javascript. Any reason why you don't have that?
2. keepalive_timeout of 70 seems to be a very very high value. Isn't it better to control it within 2/3 (as otherwise it's easy for hackers to drown the server with open files)

require 'capistrano/ext/multistage'

Данное расширение объединено с основной веткой Capistrano capistrano/capistrano@eed1a6e. Можно исключить из конфига.