tornado and supervised

gistfile1.txt

Running Tornado in production
Supervisor is a simple yet powerful client/server system to monitor and control processes on UNIX systems.

I have a number of Tornado servers that are already managed by supervisor. I have found it stable and flexible enough for most tasks.

Installing supervisor

The installation is pretty straight forward.

sudo easy_install supervisor
This will install supervisor and all the required dependencies.

Configuring supervisor

Supervisor ships with a configuration file generator. This is a good way to understand all the configuration options that supervisor supports.

Run this command if your interested in generating the default configuration.

echo_supervisord_conf > supervisord.conf
I am going to skip the default configuration file and just write one from scratch.

This is what my /etc/supervisord.conf looks like

[unix_http_server]
file=/tmp/supervisor.sock
 
[supervisord]
logfile=/var/log/supervisord.log
logfile_maxbytes=50MB
logfile_backups=10
loglevel=warn
pidfile=/var/log/supervisord.pid
nodaemon=false
minfds=1024
minprocs=200
user=nobody
childlogdir=/var/log/
 
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
 
[group:myapp]
programs=myapp_server-3001,myapp_server-3002,myapp_server-3003
 
[program:myapp_server-3001]
command=/srv/myapp/myapp_server.py --port=3001
directory=/srv/myapp/
autorestart=true
redirect_stderr=true
stdout_logfile=/srv/myapp/logs/myapp_server-3001.log
stdout_logfile_maxbytes=500MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn
 
[program:myapp_server-3002]
command=/srv/myapp/myapp_server.py --port=3002
directory=/srv/myapp/
autorestart=true
redirect_stderr=true
stdout_logfile=/srv/myapp/logs/myapp_server-3002.log
stdout_logfile_maxbytes=500MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn
 
[program:myapp_server-3003]
command=/srv/myapp/myapp_server.py --port=3003
directory=/srv/myapp/
autorestart=true
redirect_stderr=true
stdout_logfile=/srv/myapp/logs/myapp_server-3003.log
stdout_logfile_maxbytes=500MB
stdout_logfile_backups=50
stdout_capture_maxbytes=1MB
stdout_events_enabled=false
loglevel=warn
As you see I start 3 instances of my application on port 3000, 3001, and 3002. It is highly recommended to front your tornado servers with nginx.

Running Supervisor

The two main scripts that you need to use are supervisord and supervisorctl.

supervisord is the backend script that manages the processes and logs. It is also responsible for ensuring that the the processes are up and running.

supervisorctl is the control script that you can use to talk to multiple supervisord daemons. You can use this script to start multiple services or groups.

Some important commands

Basic commands

start Starts processes
stop Stops processes
restart Restarts processes
status Status of a process
The not so basic commands

update Reloads the configuration file
reload Restarts the supervisord daemon
add Adds processes/groups
remove Removes processes/groups
tail Start tailing a log file