Gracefully restarting Sidekiq on Elasticbeanstalk, only after it's done with the running jobs. Tested on: 64bit Amazon Linux 2016.09 v2.3.0 running Ruby 2.3 (Puma). Thanks to ssaunier for the original gist! https://gist.github.com/ssaunier/44bbebb9c0fa01953860

sidekiq.config

# Sidekiq interaction and startup script
commands:
  create_post_dir:
    command: "mkdir -p /opt/elasticbeanstalk/hooks/appdeploy/post"
    ignoreErrors: true
files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/50_restart_sidekiq.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      . /opt/elasticbeanstalk/support/envvars

      EB_APP_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
      EB_APP_PID_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)
      EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)

      . $EB_SUPPORT_DIR/envvars
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      SIDEKIQ_PID=$EB_APP_PID_DIR/sidekiq.pid
      SIDEKIQ_CONFIG=$EB_APP_DEPLOY_DIR/config/sidekiq.yml
      SIDEKIQ_LOG=$EB_APP_DEPLOY_DIR/log/sidekiq.log

      . /opt/elasticbeanstalk/support/envvars.d/sysenv

      sleep 10

      cd $EB_APP_DEPLOY_DIR

      echo "starting sidekiq"
      su -s /bin/bash -c "bundle exec sidekiq \
        -e $RACK_ENV \
        -P $SIDEKIQ_PID \
        -C $SIDEKIQ_CONFIG \
        -L $SIDEKIQ_LOG \
        -d" $EB_APP_USER

  "/opt/elasticbeanstalk/hooks/appdeploy/pre/03_mute_sidekiq.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash
      . /opt/elasticbeanstalk/support/envvars

      EB_APP_DEPLOY_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
      EB_APP_PID_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k app_pid_dir)
      EB_APP_USER=$(/opt/elasticbeanstalk/bin/get-config container -k app_user)
      EB_SCRIPT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k script_dir)
      EB_SUPPORT_DIR=$(/opt/elasticbeanstalk/bin/get-config container -k support_dir)

      . $EB_SUPPORT_DIR/envvars
      . $EB_SCRIPT_DIR/use-app-ruby.sh

      SIDEKIQ_PID=$EB_APP_PID_DIR/sidekiq.pid

      cd $EB_APP_DEPLOY_DIR

      if [ -f $SIDEKIQ_PID ]
      then
        echo "waiting for sidekiq to finish busy jobs"
        su -s /bin/bash -c "bundle exec rake sidekiq:wait" $EB_APP_USER
        echo "shutting down sidekiq"
        su -s /bin/bash -c "kill -TERM `cat $SIDEKIQ_PID`" $EB_APP_USER
        su -s /bin/bash -c "rm -rf $SIDEKIQ_PID" $EB_APP_USER
      fi

sidekiq.rake

namespace :sidekiq do
  require 'sidekiq/api'

  desc "Wait until 'busy' queue is finished"
  task wait: :environment do
    Sidekiq::ProcessSet.new.each(&:quiet!)
    sleep(1) until finished?
  end

  private

  def finished?
    ps = Sidekiq::ProcessSet.new
    return ps.size == 0 || ps.detect { |process| process['busy'] == 0 }
  end
end

When running this script with multiple EC2 instances, after a deploy with Elastic Beanstalk one instance often gets stuck on quiet
Any thoughts on how to prevent this?

@andrewlynch Is your problem solved? If it's solved, can you tell me how you did it?

@andrewlynch Did you put the files in the right directories? This isn't supposed to happen because sidekiq:wait is executed pre-deploy
I'm no longer working with Elasticbeanstalk so I'm not sure if it's working the same way it used to

@maxdbn sidekiq/sidekiq#4336

@rokumatsumoto If I remember correctly I only quiet the processes so they won't get any new jobs so I can restart sidekiq without losing anything. When it's restarted they should work as usual, the issue you sent is not relevant :)

Confirmed working on Passenger with Ruby 2.6.0 running on 64bit Amazon Linux/2.7.1 on 23/11/19.

thanks

This script only works when you have a single worker. If you have a multiple workers with rolling deployments, it will fail. It will deploy 1 server at a time and every time it will quite all of the process even which are already deployed which causes the issue mentioned by andrewlynch .