Install and use Supervisord with AWS Elastic Beanstalk, Symfony 2 and RabbitMq Bundle

Readme.md

• Place the supervisord.conf under .ebextensions/supervisor/

• Place the supervisor.config under .ebextensions/

• Place the data_import_consumer.conf under .ebextensions/supervisor/

data_import_consumer.conf

[program:data_import_consumer]
process_name= %(process_num)s
command= /usr/bin/php /var/app/current/api/app/console rabbitmq:multiple-consumer -m 30 import_data
autostart=true
autorestart=unexpected
stderr_logfile=/var/log/app/rabbitmq/data_import_consumer.err.log
stdout_logfile=/var/log/app/rabbitmq/data_import_consumer.out.log
numprocs=3
exitcodes=1
startsecs=0

supervisor.config

commands:
  11install_supervisor:
    test : '[ ! /etc/supervisor ] && echo "supervisor not installed"'
    command: "easy_install supervisor"


container_commands:
    14insert_supervisor_main_config:
        command: ". /opt/elasticbeanstalk/support/envvars && cat .ebextensions/supervisor/supervisord.conf > /etc/supervisor/supervisord.conf"
        leader_only: true
    15insert_supervisor_main_config_to_etc:
        command: ". /opt/elasticbeanstalk/support/envvars && cat .ebextensions/supervisor/supervisord.conf > /etc/supervisord.conf"
        leader_only: true
    16insert_data_import_supervisor_config:
        command: ". /opt/elasticbeanstalk/support/envvars && cat  .ebextensions/supervisor/data_import_consumer.conf > /etc/supervisor/conf.d/data_import_consumer.conf"
        leader_only: true

files:
    "/opt/elasticbeanstalk/hooks/appdeploy/post/490-supervisor_install.sh":
        mode: "000755"
        owner: root
        group: root
        content: |
            #!/usr/bin/env bash
            . /opt/elasticbeanstalk/support/envvars

            if [ ! -f /usr/local/bin/supervisord ]; then
                echo "install supervisor and create directories"
                easy_install supervisor
            else
                echo "supervisor already installed"
            fi

            if [ ! -d /etc/supervisor ]; then
                mkdir /etc/supervisor
                echo "create supervisor directory"
            fi

            if [ ! -d /etc/supervisor/conf.d ]; then
                mkdir /etc/supervisor/conf.d
                echo "create supervisor configs directory"
            fi
    "/opt/elasticbeanstalk/hooks/appdeploy/post/500-supervisor_update.sh":
        mode: "000755"
        owner: root
        group: root
        content: |
            #!/usr/bin/env bash
            . /opt/elasticbeanstalk/support/envvars

            if ps aux | grep -q "[/]usr/local/bin/supervisord"; then
            echo "supervisor is running"
            else
            echo "start supervisor"
            /usr/bin/python /usr/local/bin/supervisord --pidfile /var/run/supervisord.pid
            fi

            supervisorctl reread
            supervisorctl update

            cd /var/app/current/api
            app/console rabbitmq:setup-fabric --env=$SYMFONY_ENV

            echo "$(date +'%Y%m%d %T') Check for supervisor update" >> /var/log/directory-hooks-executor.log

supervisord.conf

[unix_http_server]
file=/tmp/supervisor.sock   ; (the path to the socket file)

[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)
environment=SYMFONY_ENV=prod

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

[include]
files = /etc/supervisor/conf.d/*.conf

[inet_http_server]
port = 9000
username = user
password = pw

When i try to use this in elastic beanstalk i get this error:

The configuration file __MACOSX/.ebextensions/._supervisor.config in application version XXXXX contains invalid YAML or JSON. YAML exception: Invalid Yaml: unacceptable character '�' (0x0) special characters are not allowed in "", position 0, JSON exception: Invalid JSON: Unexpected character (�) at position 0.. Update the configuration file.

I copied the code from the above code in the RAW format, so shouldn't have had any special characters in it.

Any thoughts?

Thanks!

@mellis3 Sorry for the late answer but unfortunately Github doesn't send notifications for comments here.
Do you use any IDE, maybe the spaces were converted to tabs or something else. The script was working under ubuntu but it's also some time ago.
The only thing I can suggest is to debug it step by step. Comment out some parts and you'll find the wrong character.

With the new supervisor (version 3.0+ I think), the folder structures change. There is no longer a supervisor folder in /etc, so I had to change the following commands as such:
04_insert_supervisor_main_config: command: ". /opt/elasticbeanstalk/support/envvars && cat .ebextensions/supervisor/supervisord.conf > /etc/supervisord.conf" leader_only: true 05_insert_supervisor_main_config_to_etc: command: ". /opt/elasticbeanstalk/support/envvars && cat .ebextensions/supervisor/supervisord.conf > /etc/supervisord.conf" leader_only: true 06_insert_push_listener_supervisor_config: command: ". /opt/elasticbeanstalk/support/envvars && cat .ebextensions/supervisor/push_listener.conf > /etc/supervisor.conf" leader_only: true

Otherwise, everything else worked flawlessly. Thank you for this!

@mellis3 You probably figured it out already by now, but maybe this will help someone else. This error is caused because of __MACOSX folders that are created automatically by MacOSX and shouldn't be in the zip you upload to the Elastic Beanstalk. To fix this, try:

zip -d Archive.zip "__MACOSX*"

(where Archive.zip is points to your zip)

Thank you, this was very useful!

Thank you this gist post!.

please, how can i stop supervisor on deploy wity symfony ?

@kprosper Maybe you could try to create a pre-deploy file i.e. /opt/elasticbeanstalk/hooks/appdeploy/pre/100-stop-supervisor.sh which stops supervisor.
Like I did it in supervisor.config for updating supervisor.

I am getting this error

/bin/sh: /opt/elasticbeanstalk/support/envvars: No such file or directory

in amazon linux 2.

Any Replace of the line?

@danishmirza

Hi danishmirza,
unfortunately this gist is very old. I think in the meantime AWS changed the structure in their Elastic Beanstalk instances. Do you have a chance to find out the directory for the environment variables on AWS instance?
Then you need to replace /opt/elasticbeanstalk/support/envvars with the current path.

Or you need to create the envvars file first.
See https://aws.amazon.com/premiumsupport/knowledge-center/view-environment-properties-elastic-beanstalk/

@danishmirza
I just found an article in stackoverflow for Amazon Linux 2 instances. It's a little bit different from Amazon 1 instances.
You need to do something like this:

export $(cat /opt/elasticbeanstalk/deployment/env | xargs)

See https://stackoverflow.com/questions/63280543/aws-eb-deploy-doesnt-load-environment-variables

Good morning everybody, thank you for your good work. please I want to ask how can i Restart supervisor main process which has started by /usr/bin/python /usr/local/bin/supervisord --pidfile /var/run/supervisord.pid. thank you for your answer.

Hi @kprosper, you can create a new pre- or post-deploy script (see supervisor.config for examples)
In this script you can try to run service supervisor restart or
supervisorctl restart <name>

See here: https://www.onurguzel.com/supervisord-restarting-and-reloading/

For those struggling to make it running in 2021: https://gist.github.com/johnyvelho/6a457d5e0500ca0c1faa2e6c5eab7ced

@johnyvelho Cool, thanx for the update and your work.