Steps with explanations to set up a server using:
- Virtualenv
- Virtualenvwrapper
- Django
- Gunicorn
- Nginx
- Supervisor
Nginx will face the outside world. It will serve media files (images, CSS, etc) directly from the file system. However, it can't talk directly to Django applications; it needs something that will run the application, feed it requests from the web, and return responses.
That's Gunicorn's job. Gunicorn will create a Unix socket, and serve responses to nginx via the wsgi protocol - the socket passes data in both directions:
The outside world <-> Nginx <-> The socket <-> Gunicorn
All this family will live into a Virtualenv. Already wondered why Virtualenv is so useful when you develop Pythons's applications? Continue to read and you will understand.
If you don't already use a Virtualenv, you really should consider using it. In a nutshell, the fact is that your operating system have a lot of core modules which depends of Python. That rule is especially right if you use Ubuntu. So, do you really need to broke up something existant to create somethin new? Well, some guys could answer you "Yes". I'll teach you how to say "No" to them ! Virtualenv will create a dedicated virtual environment with his python binary as well as his own modules. With that method, your Virtualenv will be isolated from the rest of the system.
If you use Pip, and I clearly recommand to use it, Virtualenv will simplify your life, because if you play well, you only have to do
pip freeze > requirements.txt
to print a list of all the modules needed by your project into a file. When you will deploy it, you will be able to install all those module with a
pip install -r requirements.txt
What ? I didn't explained how to use Virtualenv? Well, indeed I lied. We won't be using Virtualenv directly. We will use a wrapper. Virtualenvwrapper to be right. Original, isn't it? Virtualenvwrapper will allow you to easily create virtualenv and switching between them.
Let's get started ! To install virtualenvwrapper, install it with Pip, apt-get, whatever:
pip install virtualenvwrapper
Virtualenvwrapper is now installed? Let's play with him. First, you will need to modify your .bashrc with your favourite text editor. Add this lines at the and of your .bashrc
export WORKON_HOME=~/.virtualenvs export PROJECT_HOME=/path/to/your/project/home source /usr/local/bin/virtualenvwrapper.sh
WORKON_HOME is required but PROJECT_HOME is needless. Don't forget to always put ALL the export's directives before sourcing your file. Seems legit for you? Here, have my like !
Now, let's activate our .bashrc. If you are a UNIX guru, you have problably already did it before I explained it. If not, here is the command
source .bashrc
If all is good, your stdout will be sourced byt the output of the Virtualenvwrapper creating script. You can verify it by searching a .virtualenvs folder is your home directory.
Now, let's ceate our virtualenv for our project. You can name it whatever your want and even if you don't remember, just go into your .virtualenvs folder and the name of your virtualenv will be one of the folder. The command for creating one is
mkvirtualenv nameofyourproject
The command for working on a specified virtualenv is
workon nameofyourproject
The command for deactivate a specified virtualenv is
deactivate nameofyourproject
And finally the command to remove a specified virtualenv is
rmvirtualenv nameofyourproject
That's all the command you need to remember. If the workon command has successfully created a virtualenv, your should see the name of your project between parentheses in the left of your username@nameofyourcomputer. Something like that
(nameofyourproject)username@nameofyourcomputer
All is good, Okey, let's move to the next of the story.
! REMEMBER TO DO ALL THE NEXT PART IN YOUR VIRTUALENV. ONLY SUPERVISOR WILL BE INSTALLED OUTSIDE YOUR VIRTUALENV !
I'm assuming you are using Django 1.4.x. Everything should be fine from our right for Django 1.5. Anayway, if you created your project in 1.4, it should have automatically created a wsgi module. If you're using an earlier version, you will have to find a Django wsgi module for your project. Here is mine modified for exemple
import os import sys root = os.path.join(os.path.dirname(__file__), '..') sys.path.insert(0, root) os.environ['DJANGO_SETTINGS_MODULE'] = 'nameofyourproject.settings' import django.core.handlers.wsgi application = django.core.handlers.wsgi.WSGIHandler() # This application object is used by any WSGI server configured to use this # file. This includes Django's development server, if the WSGI_APPLICATION # setting points here. from django.core.wsgi import get_wsgi_application application = get_wsgi_application() # Apply WSGI middleware here. # from helloworld.wsgi import HelloWorldApplication # application = HelloWorldApplication(application)
Note that I'm also assuming a Django 1.4 project structure, in which you see paths like:
/path/to/your/project/project/
(i.e. it creates nested directories with the name of your project). Adjust the examples if you using an earlier Django.
It will also be helpful if you are in your Django project's directory. If you don't have one ready, just create a directory for now.
I'll call your domain domain.tld. Substitute your own FQDN or IP address.
Throughout, I'm using the port (:8000) for tests because it is the same port as Django's dev web server. The default port for http (:80) will be used for real deployment. You can use whatever port you want of course, but if you don't know what NAT is, don't do it.
As we said earlier, Gunicorn will serve the core of our application, just like the Django's development web server would do it. Let's install it
pip install gunicorn
Create a file called myapp.py:
def app(environ, start_response):
data = "Hello, World!\n"
start_response("200 OK", [
("Content-Type", "text/plain"),
("Content-Length", str(len(data)))
])
return iter([data])
Run:
gunicorn -w 4 myapp:app
This should serve a hello world message directly to the browser on port 8000. Visit:
http://127.0.0.1:8000
to check.
Now we want gunicorn to do the same thing, but to run a Django site instead of the test.py module.
But first, make sure that your project actually works! Now you need to be in your Django project directory.
python manage.py runserver 0.0.0.0:8000
Now run it using gunicorn:
gunicorn nameofyourapp.wsgi:app
Point your browser at the server; if the site appears, it means gunicorn can serve your Django application from your virtualenv. Media/static files may not be served properly, but don't worry about that.
Now normally we won't have the browser speaking directly to gunicorn: nginx will be the go-between.
The version of Nginx from Debian stable and Ubuntu is rather old. We'll install from backports for Debian and PPA for Ubuntu.
Debian:
sudo nano /etc/apt/sources.list # edit the sources list
Add:
# backports deb http://backports.debian.org/debian-backports squeeze-backports main
Run:
sudo apt-get -t squeeze-backports install nginx # install nginx
For Ubuntu:
sudo -s nginx=stable # use nginx=development for latest development version add-apt-repository ppa:nginx/$nginx apt-get update apt-get install nginx
For both:
sudo /etc/init.d/nginx start # start nginx
And now check that the server is serving by visiting it in a web browser on port 80 - you should get a message from nginx: "Welcome to nginx!"
Create a file called nginx.conf, and put this in it:
server {
# the port your site will be served on
listen 80;
# the domain name it will serve for
server_name .domain.tld ip.adress; # substitute by your FQDN and machine's IP address
charset utf-8;
#Max upload size
client_max_body_size 75M; # adjust to taste
# Django media
location /media {
alias /var/www/path/to/your/project/media; # your Django project's media files
}
location /assets {
alias /var/www/path/to/your/project/static; # your Django project's static files
}
# Finally, send all non-media requests to the Django server.
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
Move your file to /etc/nginx/sites-enabled so nginx can see it:
sudo mv /path/to/your/project/nginx.conf /etc/nginx/sites-enabled/
Restart nginx:
sudo /etc/init.d/nginx restart
Check that media files are being served correctly:
Add an image called media.png to the /path/to/your/project/project/media directory
Visit
http://domain.tld:80/media/media.png
If this works, you'll know at least that nginx is serving files correctly.
Let's run our Django application:
gunicorn nameofyourapp.wsgi:app
Now gunicorn and nginx should be serving up your Django application.
The last step is to make it all happen automatically at system startup time.
To do this, we will install a programm called supervisor:
sudo pip install supervisor
Sudo in front of your command is important, because it means that our supervisor programm will be installed in the system and not in a virtualenv.
Now, we will create a bash file to execute some commands such as activate our virtualenv and browsing to our directory. So create a nameoryourproject.sh file:
#!/bin/bash
WORKING_DIR=/path/to/your/project
ACTIVATE_PATH=/path/to/your/virtualenv/bin/activate
cd ${WORKING_DIR}
source ${ACTIVATE_PATH}
exec $@
Next, we create a file named supervisord.conf and we put it in /etc/
; Sample supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html
;
; Note: shell expansion ("~" or "$HOME") is not supported. Environment
; variables can be expanded using this syntax: "%(ENV_HOME)s".
[unix_http_server]
file=/tmp/supervisor.sock ; (the path to the socket file)
chmod=0700 ; socket file mode (default 0700)
chown=root:root ; socket file uid:gid owner
;username=noname ; (default is no username (open server))
;password=noname ; (default is no password (open server))
;[inet_http_server] ; inet (TCP) server disabled by default
;port=127.0.0.1:9001 ; (ip_address:port specifier, *:port for all iface)
;username=noname ; (default is no username (open server))
;password=noname ; (default is no password (open server))
[supervisord]
logfile=/var/log/supervisord/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=true ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
;umask=022 ; (process file creation umask;default 022)
user=root ; (default is current user, required if root)
;identifier=supervisor ; (supervisord identifier, default is 'supervisor')
;directory=/tmp ; (default is not to cd during start)
;nocleanup=true ; (don't clean up tempfiles at start;default false)
;childlogdir=/tmp ; ('AUTO' child log dir, default $TEMP)
;environment=KEY=value ; (key value pairs to add to environment)
;strip_ansi=false ; (strip ansi escape codes in logs; def. false)
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
;username=www-data ; should be same as http_username if set
;password=www-data ; should be same as http_password if set
;prompt=mysupervisor ; cmd line prompt (default "supervisor")
;history_file=~/.sc_history ; use readline history if available
; The below sample program section shows all possible program subsection values,
; create one or more 'real' program: sections to be able to control them under
; supervisor.
;[program:theprogramname]
;command=/bin/cat ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1 ; number of processes copies to start (def 1)
;directory=/tmp ; directory to cwd to before exec (def no cwd)
;umask=022 ; umask for process (default None)
;priority=999 ; the relative start priority (default 999)
;autostart=true ; start at supervisord start (default: true)
;autorestart=unexpected ; whether/when to restart (default: unexpected)
;startsecs=1 ; number of secs prog must stay running (def. 1)
;startretries=3 ; max # of serial start failures (default 3)
;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
;stopsignal=QUIT ; signal used to kill process (default TERM)
;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
;stopasgroup=false ; send stop signal to the UNIX process group (default false)
;killasgroup=false ; SIGKILL the UNIX process group (def false)
;user=chrism ; setuid to this UNIX account to run the program
;redirect_stderr=true ; redirect proc stderr to stdout (default false)
;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10)
;stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
;stdout_events_enabled=false ; emit events on stdout writes (default false)
;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups=10 ; # of stderr logfile backups (default 10)
;stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
;stderr_events_enabled=false ; emit events on stderr writes (default false)
;environment=A=1,B=2 ; process environment additions (def no adds)
;serverurl=AUTO ; override serverurl computation (childutils)
; The below sample eventlistener section shows all possible
; eventlistener subsection values, create one or more 'real'
; eventlistener: sections to be able to handle event notifications
; sent by supervisor.
;[eventlistener:theeventlistenername]
;command=/bin/eventlistener ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1 ; number of processes copies to start (def 1)
;events=EVENT ; event notif. types to subscribe to (req'd)
;buffer_size=10 ; event buffer queue size (default 10)
;directory=/tmp ; directory to cwd to before exec (def no cwd)
;umask=022 ; umask for process (default None)
;priority=-1 ; the relative start priority (default -1)
;autostart=true ; start at supervisord start (default: true)
;autorestart=unexpected ; whether/when to restart (default: unexpected)
;startsecs=1 ; number of secs prog must stay running (def. 1)
;startretries=3 ; max # of serial start failures (default 3)
;exitcodes=0,2 ; 'expected' exit codes for process (default 0,2)
;stopsignal=QUIT ; signal used to kill process (default TERM)
;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
;stopasgroup=false ; send stop signal to the UNIX process group (default false)
;killasgroup=false ; SIGKILL the UNIX process group (def false)
;user=chrism ; setuid to this UNIX account to run the program
;redirect_stderr=true ; redirect proc stderr to stdout (default false)
;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10)
;stdout_events_enabled=false ; emit events on stdout writes (default false)
;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups ; # of stderr logfile backups (default 10)
;stderr_events_enabled=false ; emit events on stderr writes (default false)
;environment=A=1,B=2 ; process environment additions
;serverurl=AUTO ; override serverurl computation (childutils)
; The below sample group section shows all possible group values,
; create one or more 'real' group: sections to create "heterogeneous"
; process groups.
;[group:thegroupname]
;programs=progname1,progname2 ; each refers to 'x' in [program:x] definitions
;priority=999 ; the relative start priority (default 999)
; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files *cannot*
; include files themselves.
;[include]
;files = relative/directory/*.ini
[program:nameofyourprogram]
directory = /path/of/your/project/
user = www-data
command = /path/to/your/nameofyourproject.sh gunicorn nameofyourproject.wsgi:application
stdout_logfile = /var/log/supervisord/access.log
stderr_logfile = /var/log/supervisord/error.log
After, we will create a file which will ask supervisord to boot at the startup.
Create a file named supervisord:
# Supervisord auto-start
#
# description: Auto-starts supervisord
# processname: supervisord
# pidfile: /var/run/supervisord.pid
SUPERVISORD=/usr/local/bin/supervisord
SUPERVISORD_ARGS='-c /etc/supervisord.conf'
SUPERVISORCTL=/usr/local/bin/supervisorctl
case $1 in
start)
echo -n "Starting supervisord: "
$SUPERVISORD $SUPERVISORD_ARGS
echo
;;
stop)
echo -n "Stopping supervisord: "
$SUPERVISORCTL shutdown
echo
;;
restart)
echo -n "Stopping supervisord: "
$SUPERVISORCTL shutdown
echo
echo -n "Starting supervisord: "
$SUPERVISORD $SUPERVISORD_ARGS
echo
;;
esac
Make it executable
chmod +x supervisord
Move it to /etc/init.d
mv supervisord /etc/init.d/
Make it boot at the start
sudo update-rc.d supervisord defaults
Reboot your system and your website should live through the ages.
Thanks a lot for good explanation of how does gunicorn+nginx + supervisor in django project