NGINX+SPDY with Unicorn. True Zero-Downtime unless migrations. Best practices.

0. nginx_setup.sh

# Nginx+Unicorn best-practices congifuration guide. Heartbleed fixed.
# We use latest stable nginx with fresh **openssl**, **zlib** and **pcre** dependencies.
# Some extra handy modules to use: --with-http_stub_status_module --with-http_gzip_static_module
#
# Deployment structure
#
# SERVER:
#  /etc/init.d/nginx                             (1. nginx)
#  /home/app/public_html/app_production/current  (Capistrano directory)
#
# APP:
#  config/server/production/nginx.conf           (2. nginx.conf)
#  config/server/production/nginx_host.conf      (3. nginx_host.conf)
#  config/server/production/nginx_errors.conf    (4. nginx_errors.conf)
#  config/deploy.rb                              (5. deploy.rb)
#  config/deploy/production.rb                   (6. production.rb)
#  config/server/production/unicorn.rb           (7. unicorn.rb)
#  config/server/production/unicorn_init.sh      (8. unicorn_init.sh)






cd /usr/src
wget http://nginx.org/download/nginx-1.5.13.tar.gz
tar xzvf ./nginx-1.5.13.tar.gz && rm -f ./nginx-1.5.13.tar.gz

wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.32.tar.gz
tar xzvf pcre-8.32.tar.gz && rm -f ./pcre-8.32.tar.gz

wget http://www.openssl.org/source/openssl-1.0.1g.tar.gz
tar xzvf openssl-1.0.1g.tar.gz && rm -f openssl-1.0.1g.tar.gz

cd nginx-1.5.13
./configure --prefix=/opt/nginx --with-pcre=/usr/src/pcre-8.32 --with-openssl-opt=no-krb5 --with-openssl=/usr/src/openssl-1.0.1g --with-http_ssl_module --with-http_spdy_module --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --with-http_stub_status_module --with-http_gzip_static_module

make && make install

mkdir /tmp/client_body_temp
mkdir /opt/nginx/ssl_certs 

echo "include /home/app/public_html/app_production/current/config/server/production/nginx.conf;" > /opt/nginx/conf/nginx.conf 

vim /etc/init.d/nginx # see the config below
chmod +x /etc/init.d/nginx && update-rc.d -f nginx defaults

1. nginx

#! /bin/sh

### BEGIN INIT INFO
# Provides:          nginx
# Required-Start:    $all
# Required-Stop:     $all
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: starts the nginx web server
# Description:       starts nginx using start-stop-daemon
### END INIT INFO
#
# /etc/init.d/nginx


PATH=/opt/nginx/sbin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/opt/nginx/sbin/nginx
NAME=nginx
DESC=nginx

test -x $DAEMON || exit 0

# Include nginx defaults if available
if [ -f /etc/default/nginx ] ; then
        . /etc/default/nginx
fi

set -e

case "$1" in
  start)
        echo -n "Starting $DESC: "
        start-stop-daemon --start --quiet --pidfile /opt/nginx/logs/$NAME.pid \
                --exec $DAEMON -- $DAEMON_OPTS
        echo "$NAME."
        ;;
  stop)
        echo -n "Stopping $DESC: "
        start-stop-daemon --stop --quiet --pidfile /opt/nginx/logs/$NAME.pid \
                --exec $DAEMON
        echo "$NAME."
        ;;
  restart|force-reload)
        echo -n "Restarting $DESC: "
        start-stop-daemon --stop --quiet --pidfile \
                /opt/nginx/logs/$NAME.pid --exec $DAEMON
        sleep 1
        start-stop-daemon --start --quiet --pidfile \
                /opt/nginx/logs/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS
        echo "$NAME."
        ;;
  reload)
          echo -n "Reloading $DESC configuration: "
          start-stop-daemon --stop --signal HUP --quiet --pidfile     /opt/nginx/logs/$NAME.pid \
              --exec $DAEMON
          echo "$NAME."
          ;;
      *)
            N=/etc/init.d/$NAME
            echo "Usage: $N {start|stop|restart|reload|force-reload}" >&2
            exit 1
            ;;
    esac

    exit 0

2. nginx.conf

# Nginx main block configuration file.
# The most important directives here are ssl_protocols and ssl_ciphers
# Keep nginx configuration in repo, then just include it in /opt/nginx/conf/nginx.conf
# And DDoS prevent attack with directive limit_req_zone (limit 10 request/sec from 1 IP address)
# then it enables in the server block by "limit_req zone=one".
#
# config/server/production/nginx.conf


user                          app;
worker_processes              2;
worker_priority               -5;
timer_resolution              100ms;

error_log                     logs/nginx.error.log;

events {
    use                       epoll;
    worker_connections        2048;
}


http {
    client_max_body_size      25m;
    client_body_buffer_size   128k;
    client_body_temp_path     /tmp/client_body_temp;

    include                   mime.types;
    default_type              application/octet-stream;
    server_tokens             off;
    sendfile                  on;
    tcp_nopush                on;
    tcp_nodelay               on;
    keepalive_timeout         70;

    gzip                      on;
    gzip_http_version         1.1;
    gzip_disable              "msie6";
    gzip_vary                 on;
    gzip_min_length           1100;
    gzip_buffers              64 8k;
    gzip_comp_level           3;
    gzip_proxied              any;
    gzip_types                text/plain text/css application/x-javascript text/xml application/xml;

    ssl_certificate           /opt/nginx/ssl_certs/server.crt;
    ssl_certificate_key       /opt/nginx/ssl_certs/server.key;
    ssl_session_timeout       15m;
    ssl_protocols             SSLv3 TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_session_cache         shared:SSL:10m;
    ssl_stapling              on;

    add_header                Strict-Transport-Security "max-age=16070400; includeSubdomains";
    add_header                X-Frame-Options DENY;

    limit_req_zone            $binary_remote_addr zone=one:10m rate=50r/s;

    include                   /home/app/public_html/app_production/current/config/server/nginx_host.conf;
}

3. nginx_host.conf

# Nginx server block configuration with proxy_pass to Unicorn upstream
# We use full-SSL site with web-server redirection, no mess with Rails application redirection
#
# config/server/production/nginx_host.conf


upstream unicorn {
  server              unix:/tmp/unicorn.production.sock fail_timeout=0;
}


server {
  listen              80;
  server_name         server.com;
  rewrite             ^(.*) https://$host$1 permanent;

  location ~ \.(php|html)$ {
    deny              all;
  }

  access_log          /dev/null;
  error_log           /dev/null;
}


server {
  ssl                 on;
  listen              443 spdy ssl;
  server_name         server.com;
  root                /home/app/public_html/app_production/current/public;
  try_files           $uri /system/maintenance.html @unicorn;

  location @unicorn {
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_set_header  Host $http_host;
    proxy_redirect    off;
    proxy_pass        http://unicorn;
    limit_req         zone=one burst=15;

    access_log        /dev/null;
    error_log         logs/unicorn.error.log;
  }

  location ~ ^/(assets|images|javascripts|stylesheets|swfs|system)/ {
    gzip_static       on;
    expires           max;
    add_header        Cache-Control public;
    add_header        Last-Modified "";
    add_header        ETag "";
  }

  include             /home/app/public_html/app_production/current/config/server/production/nginx_errors.conf;

  access_log          /dev/null;
  error_log           /dev/null;
}

4. nginx_errors.conf

# nginx configuration piece to handle errorrs
#
# config/server/production/nginx_errors.conf

error_page          500 502 504  /500.html;
error_page          503 @503;

location = /50x.html {
    root            html;
}

location = /404.html {
    root            html;
}

location @503 {
  error_page 405 = /system/maintenance.html;
  if (-f $document_root/system/maintenance.html) {
    rewrite         ^(.*)$ /system/maintenance.html break;
  }
  rewrite           ^(.*)$ /503.html break;
}

if ($request_method !~ ^(GET|HEAD|PUT|POST|DELETE|OPTIONS)$ ){
  return            405;
}

if (-f $document_root/system/maintenance.html) {
  return            503;
}

location ~ \.(php|html)$ {
  return            405;
}

5. deploy.rb

# Capistrano configuration. Now with TRUE zero-downtime unless DB migration.
# 
# require 'new_relic/recipes'         - Newrelic notification about deployment
# require 'capistrano/ext/multistage' - We use 2 deployment environment: staging and production.
# set :deploy_via, :remote_cache      - fetch only latest changes during deployment
# set :normalize_asset_timestamps     - no need to touch (date modification) every assets
# "deploy:web:disable"                - traditional maintenance page (during DB migrations deployment)
# task :restart                       - Unicorn with preload_app should be reloaded by USR2+QUIT signals, not HUP
#
# http://unicorn.bogomips.org/SIGNALS.html
# "If “preload_app” is true, then application code changes will have no effect; 
# USR2 + QUIT (see below) must be used to load newer code in this case"
# 
# config/deploy.rb


require 'bundler/capistrano'
require 'capistrano/ext/multistage'
require 'new_relic/recipes'

set :stages,                     %w(staging production)
set :default_stage,              "staging"

set :scm,                        :git
set :repository,                 "..."
set :deploy_via,                 :remote_cache
default_run_options[:pty]        = true

set :application,                "app"
set :use_sudo,                   false
set :user,                       "app"
set :normalize_asset_timestamps, false


before "deploy",                 "deploy:delayed_job:stop"
before "deploy:migrations",      "deploy:delayed_job:stop"

after  "deploy:update_code",     "deploy:symlink_shared"
before "deploy:migrate",         "deploy:web:disable", "deploy:db:backup"

after  "deploy",                                      "newrelic:notice_deployment", "deploy:cleanup", "deploy:delayed_job:restart"
after  "deploy:migrations",      "deploy:web:enable", "newrelic:notice_deployment", "deploy:cleanup", "deploy:delayed_job:restart"


namespace :deploy do

  %w[start stop].each do |command|
    desc "#{command} unicorn server"
    task command, :roles => :app, :except => { :no_release => true } do
      run "#{current_path}/config/server/#{rails_env}/unicorn_init.sh #{command}"
    end
  end

  desc "restart unicorn server"
  task :restart, :roles => :app, :except => { :no_release => true } do
    run "#{current_path}/config/server/#{rails_env}/unicorn_init.sh upgrade"
  end


  desc "Link in the production database.yml and assets"
  task :symlink_shared do
    run "ln -nfs #{deploy_to}/shared/config/database.yml #{release_path}/config/database.yml"
  end

  namespace :delayed_job do 
    desc "Restart the delayed_job process"
    task :restart, :roles => :app, :except => { :no_release => true } do
      run "cd #{current_path}; RAILS_ENV=#{rails_env} bundle exec script/delayed_job restart" rescue nil
    end
    desc "Stop the delayed_job process"
    task :stop, :roles => :app, :except => { :no_release => true } do
      run "cd #{current_path}; RAILS_ENV=#{rails_env} bundle exec script/delayed_job stop" rescue nil
    end
  end


  namespace :db do
    desc "backup of database before migrations are invoked"
    task :backup, :roles => :db, :only => { :primary => true } do
      filename = "#{deploy_to}/shared/db_backup/#{stage}_db.#{Time.now.utc.strftime("%Y-%m-%d_%I:%M")}_before_deploy.gz"
      text = capture "cat #{deploy_to}/current/config/database.yml"
      yaml = YAML::load(text)["#{stage}"]
    
      on_rollback { run "rm #{filename}" }
      run "mysqldump --single-transaction --quick -u#{yaml['username']} -h#{yaml['host']} -p#{yaml['password']} #{yaml['database']} | gzip -c > #{filename}"
    end
  end


  namespace :web do
    desc "Maintenance start"
    task :disable, :roles => :web do
      on_rollback { run "rm #{shared_path}/system/maintenance.html" }
      page = File.read("public/503.html")
      put page, "#{shared_path}/system/maintenance.html", :mode => 0644
    end
    
    desc "Maintenance stop"
    task :enable, :roles => :web do
      run "rm #{shared_path}/system/maintenance.html"
    end
  end

end


namespace :log do
  desc "A pinch of tail"
  task :tailf, :roles => :app do
    run "tail -n 10000 -f #{shared_path}/log/#{rails_env}.log" do |channel, stream, data|
      puts "#{data}"
      break if stream == :err
    end
  end
end

6. production.rb

# capistrano production config
#
# config/deploy/production.rb


server "8.8.8.8",                :app, :web, :db, :primary => true
set :branch,                     "production"
set :deploy_to,                  "/home/app/public_html/app_production"
set :rails_env,                  "production"

7. unicorn.rb

# Unicorn configuration file to be running by unicorn_init.sh with capistrano task
# read an example configuration before: http://unicorn.bogomips.org/examples/unicorn.conf.rb
# 
# working_directory, pid, paths - internal Unicorn variables must to setup
# worker_process 4              - is good enough for serve small production application
# timeout 30                    - time limit when unresponded workers to restart
# preload_app true              - the most interesting option that confuse a lot of us,
#                                 just setup is as true always, it means extra work on 
#                                 deployment scripts to make it correctly
# BUNDLE_GEMFILE                - make Gemfile accessible with new master
# before_fork, after_fork       - reconnect to all dependent services: DB, Redis, Sphinx etc.
#                                 deal with old_pid only if CPU or RAM are limited enough
#
# config/server/production/unicorn.rb


app_path          = "/home/app/public_html/app_production/current"

working_directory "#{app_path}"
pid               "#{app_path}/tmp/pids/unicorn.pid"
stderr_path       "#{app_path}/log/unicorn.log"
stdout_path       "#{app_path}/log/unicorn.log"

listen            "/tmp/unicorn.production.sock"
worker_processes  4
timeout           30
preload_app       true


before_exec do |server|
  ENV["BUNDLE_GEMFILE"] = "#{app_path}/Gemfile"
end


before_fork do |server, worker|
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.connection.disconnect!
  end

  if defined?(Resque)
    Resque.redis.quit
  end

  sleep 1
end


after_fork do |server, worker|
  if defined?(ActiveRecord::Base)
    ActiveRecord::Base.establish_connection
  end
  
  if defined?(Resque)
    Resque.redis           = 'localhost:6379'
  end
end

8. unicorn_init.sh

# Unicorn handle shell script
# 
# APP_ROOT, PID             - are the same as you setup above
# CMD                       - use bundle binstubs (bundle install --binstubs) to 
#                             forget about "bundle exec" stuff, run in demonize mode
#                             bin/unicorn is for Rack application (config.ru in root dir), but 
#                             bin/unicorn_rails is to use with Rails 2.3
#
# To handle "app_preload true" configuration we should use USR2+QUIT signals, not HUP!
# So we rewrite capistrano deployment scripts to manage it.
#
# config/server/production/unicorn_init.sh


#!/bin/sh
set -e
# Example init script, this can be used with nginx, too,
# since nginx and unicorn accept the same signals

TIMEOUT=${TIMEOUT-60}
APP_ROOT=/home/app/public_html/app_production/current
PID=$APP_ROOT/tmp/pids/unicorn.pid
CMD="$APP_ROOT/bin/unicorn -D -c $APP_ROOT/config/server/unicorn.rb -E production"
action="$1"
set -u

old_pid="$PID.oldbin"

cd $APP_ROOT || exit 1

sig () {
        test -s "$PID" && kill -$1 `cat $PID`
}

oldsig () {
        test -s $old_pid && kill -$1 `cat $old_pid`
}

case $action in
start)
        sig 0 && echo >&2 "Already running" && exit 0
        $CMD
        ;;
stop)
        sig QUIT && exit 0
        echo >&2 "Not running"
        ;;
force-stop)
        sig TERM && exit 0
        echo >&2 "Not running"
        ;;
restart|reload)
        sig HUP && echo reloaded OK && exit 0
        echo >&2 "Couldn't reload, starting '$CMD' instead"
        $CMD
        ;;
upgrade)
        if sig USR2 && sleep 15 && sig 0 && oldsig QUIT
        then
                n=$TIMEOUT
                while test -s $old_pid && test $n -ge 0
                do
                        printf '.' && sleep 1 && n=$(( $n - 1 ))
                done
                echo

                if test $n -lt 0 && test -s $old_pid
                then
                        echo >&2 "$old_pid still exists after $TIMEOUT seconds"
                        exit 1
                fi
                exit 0
        fi
        echo >&2 "Couldn't upgrade, starting '$CMD' instead"
        $CMD
        ;;
reopen-logs)
        sig USR1
        ;;
*)
        echo >&2 "Usage: $0 <start|stop|restart|upgrade|force-stop|reopen-logs>"
        exit 1
        ;;
esac

Very good. Please change the comment: (limit 10 request/sec from 1 IP address) to (limit 50...) about nginx limit_req_zone.

@k3NGuru по-моему, limit_req_zone был поломан и его починили только несколько месяцев назад, пробуйте стартануть nginx без него

@mikhailov Все равно не заводится система. Уже сделал как написано тут https://coderwall.com/p/8igwqa настроил как написано в данном конфиге, Вот моя папка https://github.com/k3NGuru/mlvz

2013/04/05 16:02:07 [error] 19467#0: *2 limiting requests, excess: 1.000 by zone "one", client: 192.168.91.163, server: 192.168.91.15, request: "GET / HTTP/1.1^@^Dho^@^@", host: "192.168.91.15"

192.168.91.15 - Сервер
192.168.91.163 - Клиент

SSL настроил как написано тут http://wiki.nginx.org/HttpSslModule

@k3NGuru все дело в специфичной работе встроенного модуля limit_req, добавь burst=5 для решения проблемы:

limit_req         zone=one burst=5;

Это пропустит 6 запросов в секунду, надо больше с одной айпи - увеличивай burst, либо распараллель нагрузку между несколькими Nginx серверами. Почему так, а не иначе, попытались объяснить здесь: http://www.lexa.ru/nginx-ru/msg38569.html Сегодня задан вопрос в nginx-рассылку.

Thank you for sharing this. :-)

Amazing. Thank you very much man =)

Great stuff to ease nginx usage!

A few queries/points:

1. I needed to add ' application/javascript' to gzip_types, else js was not getting compressed. Added that before application/x-javascript. Any reason why you don't have that?
2. keepalive_timeout of 70 seems to be a very very high value. Isn't it better to control it within 2/3 (as otherwise it's easy for hackers to drown the server with open files)

require 'capistrano/ext/multistage'

Данное расширение объединено с основной веткой Capistrano capistrano/capistrano@eed1a6e. Можно исключить из конфига.