Sidekiq Config for Elastic Beanstalk

sidekiq.config

# Sidekiq interaction and startup script
files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_sidekiq.sh":
    mode: "000755"
    content: |
      #!/bin/bash

      . /opt/elasticbeanstalk/hooks/common.sh
      . /opt/elasticbeanstalk/support/envvars

      set -xe

      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_CONFIG_APP_CURRENT=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
      EB_CONFIG_APP_LOGS=$(/opt/elasticbeanstalk/bin/get-config container -k app_log_dir)
      EB_CONFIG_APP_PIDS=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)

      . $EB_SCRIPT_DIR/use-app-ruby.sh

      BUNDLE=`which bundle`
      SIDEKIQ=`which sidekiq`

      cd $EB_CONFIG_APP_CURRENT

      # For 2 instances of Sidekiq this can be increased or decreased,
      # just make sure it is updated here and below. Also, if decreasing
      # these values, be sure to terminate processes at previously available
      # indices (e.g. kill sidekiq-3.pid if going from 3 to 2 processes).
      # for i in `seq 1 2`
      for i in `seq 1`
      do
        PIDFILE=$EB_CONFIG_APP_PIDS/sidekiq-$i.pid

        # Stop current Sidekiq processes
        if [ -f $PIDFILE ]
        then
          SIDEKIQ_LIVES=$(/bin/ps -o pid= -p `cat $PIDFILE`)
            if [ -z $SIDEKIQ_LIVES ]
            then
               rm -rf $PIDFILE
            else
               kill -TERM `cat $PIDFILE`
            fi
        fi

        # Boot Sidekiq process
        $BUNDLE exec $SIDEKIQ \
          -e production \
          -P $PIDFILE \
          -C $EB_CONFIG_APP_CURRENT/config/sidekiq.yml \
          -L $EB_CONFIG_APP_LOGS/sidekiq.log \
          -d
      done

  "/opt/elasticbeanstalk/hooks/appdeploy/pre/03_quiet_sidekiq.sh":
    mode: "000755"
    content: |
      #!/bin/bash

      . /opt/elasticbeanstalk/support/envvars

      EB_CONFIG_APP_PIDS=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)

      # For 2 instances of Sidekiq this can be increased or decreased,
      # just make sure it is updated here and below.
      # for i in `seq 1 2`
      for i in `seq 1`
      do
        PIDFILE=$EB_CONFIG_APP_PIDS/sidekiq-$i.pid

        # quiet any running instance
        if [ -f $PIDFILE ]
        then
          SIDEKIQ_LIVES=$(/bin/ps -o pid= -p `cat $PIDFILE`)
            if [ -z $SIDEKIQ_LIVES ]
            then
               rm -rf $PIDFILE
            else
               kill -USR1 `cat $PIDFILE`
            fi
        fi
      done

I'm new to AWS Beanstalk and looking for a way how to autoscale sidekiq workers with beanstalk.

3 Questions:

1. Does this setup run sidekiq on the app instance or on a seperated sidekiq environment?
2. Is it possible to autoscale this?
3. Is the problem from above solved?

Best, Gambo

Problem is EB is not checking stack ruby version and complains bundle with this error

Activity execution failed, because: bundler: command not found: sidekiq Install missing gem executables withbundle install(ElasticBeanstalk::ExternalInvocationError) caused by: bundler: command not found: sidekiq Install missing gem executables withbundle install(Executor::NonZeroExitStatus)

need to tell ruby version where gems are installed by adding this line or manage by environment variables.

. /opt/elasticbeanstalk/support/scripts/use-app-ruby.sh

Hope this solves issue.

@dharmdip I see that very line on the script above and it breaks anyway. with set -xe it is possible to see on the logs that it's actually setting the right ruby version. I am also facing this issue recently as it didn't use to happen.

Any other clues? Thanks!

@syntaxTerr0r @rmiadaira @phthhieu @3lackRos3 After some struggle I've come to conclusion that this issue was caused because of a bad state on the AWS Elastic Beanstalk environment.

In my case it started to happen after I modified the instance type for the environment and it had to recreate all EC2 machines. It also happened when I cloned the environment. So the solution for this was recreate a fresh environment, set the variables and redeploy the code and it got back working.

Hope that helps you too!

Sidekiq 6.0 breaking changes
The script 50_restart_sidekiq.sh will now run forever :

[root@ip-172-31-9-59 post]# ./50_restart_sidekiq.sh
starting sidekiq
`/home/webapp` is not a directory.
Bundler will use `/tmp/bundler/home/ec2-user' as your home directory temporarily.
ERROR: PID file creation was removed in Sidekiq 6.0, please use a proper process supervisor to start and manage your services
ERROR: Logfile redirection was removed in Sidekiq 6.0, Sidekiq will only log to STDOUT
ERROR: Daemonization mode was removed in Sidekiq 6.0, please use a proper process supervisor to start and manage your services
2019-10-08T18:15:49.379Z pid=29234 tid=gpaiypvxa INFO: Running in ruby 2.5.6p201 (2019-08-28 revision 67796) [x86_64-linux]
2019-10-08T18:15:49.379Z pid=29234 tid=gpaiypvxa INFO: See LICENSE and the LGPL-3.0 for licensing details.
2019-10-08T18:15:49.379Z pid=29234 tid=gpaiypvxa INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
2019-10-08T18:15:49.379Z pid=29234 tid=gpaiypvxa INFO: Booting Sidekiq 6.0.0 with redis options {:id=>"Sidekiq-server-PID-29234", :url=>nil}

This will hang your deploys : Daemonization mode was removed in Sidekiq 6.0, please use a proper process supervisor to start and manage your services

Sidekiq 5.0+
Note: The quiet signal used to be USR1 but was changed to TSTP in Sidekiq 5.0.
TSTP should be used instead of "USR1" on L79