Running a Supervised Redis to control Daphne

In a previous post we discovered that if Redis crashes and is restarted Daphne will not work properly for WebSocket connections. We discovered that it is a bug in Daphne and that using Supervisord to solve this problem might be a solution. We are going to use EventListeners in Supervisord to run a customized python script to restart Daphne if Redis crashes. First we add Redis to our supervisord configuration. Note: Remember that Redis should NOT be run in daemonized mode when running in Supervisord.

[program:Redis_Server]
command=redis-server /etc/redis/redis.conf
[...]

And we add an EventListener for our Python script that will be run on PROCESS STATE_RUNNING events from Supervisord.

[eventlistener:redis_listener]
command=python /path/to/redis_listener.py
events=PROCESS_STATE_RUNNING

Our redis-listener will listen to all events and filter out events that come from _Redis_Server. Note: This script is based on the sample from the official documentation.

import sys
import subprocess

def write_stdout(s):
    sys.stdout.write(s)
    sys.stdout.flush()

def write_file(s):
    with open("/tmp/supervisord_events.txt", 'a') as f:
    f.write(s)

def main():
    while 1:
    # transition from ACKNOWLEDGED to READY
    write_stdout('READY\n')
    # read header line and print it to stderr
    line = sys.stdin.readline()
    # read event payload and print it to stderr
    headers = dict([x.split(':') for x in line.split()])
    data = sys.stdin.read(int(headers['len']))
    headers = dict([x.split(':') for x in data.split()])
    if headers['processname'] == 'Redis_Server':
        cmd = "sudo supervisorctl -c /etc/supervisord.conf restart Daphne"
        write_file("COMMAND: " + str(cmd.split(" ")))
        process = subprocess.Popen(cmd.split(" "), stdout=subprocess.PIPE)
        out, err = process.communicate()
        write_file(out)

    # transition from READY to ACKNOWLEDGED
    write_stdout('RESULT 2\nOK')

if __name__ == '__main__':
    main()

Now whenever Redis crashes Daphne will restart when Redis is back in running state. We can test this by stopping Redis redis-cli shutdown and then read the output from the events file.

COMMAND: ['sudo', 'supervisorctl', '-c', '/etc/supervisord.conf', 'restart', 'Daphne']
Daphne: stopped
Daphne: started</pre>

Conclusion

Success! We now have a solution that can run Redis as the channel layer without fear of Daphne not being able to function properly. It should be noted that this is not a solution to the actual problem of the bug in Daphne, only a workaround.

mortenege/SupervisedRedisToControlDaphne.md

Running a Supervised Redis to control Daphne

Conclusion