Skip to content

Instantly share code, notes, and snippets.

@wojtha

wojtha/jobs_retry.rb

Created April 14, 2020 19:00
Show Gist options
  • Save wojtha/6f5c269d879c76ee7e7167abf171da62 to your computer and use it in GitHub Desktop.
Save wojtha/6f5c269d879c76ee7e7167abf171da62 to your computer and use it in GitHub Desktop.
Download ZIP
Collection of background jobs retry strategies
Raw
jobs_retry.rb
# This module collects various background job retry schedule strategies including the helper functions for the rendering
# of the retry table schedule to give better idea in which time frame the jobs are going to be run again.
#
# Inspired by https://github.com/isaacseymour/activejob-retry and
# https://github.com/mperham/sidekiq/wiki/Error-Handling#automatic-job-retry
#
module JobRetry
module_function
# Builtin default Sidekiq retry strategy.
#
# @see https://github.com/mperham/sidekiq/wiki/Error-Handling#automatic-job-retry
#
# Approximate waiting times for given attempt for the builtin Sidekiq retry logic.
#
# # | Next retry backoff | Total waiting time
# --------------------------------------------
# 0 | 0d 00h 00m 40s | 0d 00h 00m 40s
# 1 | 0d 00h 01m 08s | 0d 00h 01m 48s
# 2 | 0d 00h 01m 55s | 0d 00h 03m 43s
# 3 | 0d 00h 02m 40s | 0d 00h 06m 23s
# 4 | 0d 00h 05m 11s | 0d 00h 11m 34s
# 5 | 0d 00h 10m 46s | 0d 00h 22m 20s
# 6 | 0d 00h 24m 25s | 0d 00h 46m 45s
# 7 | 0d 00h 42m 56s | 0d 01h 29m 41s
# 8 | 0d 01h 12m 07s | 0d 02h 41m 48s
# 9 | 0d 01h 53m 16s | 0d 04h 35m 04s
# 10 | 0d 02h 49m 40s | 0d 07h 24m 44s
# 11 | 0d 04h 07m 16s | 0d 11h 32m 00s
# 12 | 0d 05h 52m 08s | 0d 17h 24m 08s
# 13 | 0d 08h 02m 48s | 1d 01h 26m 56s
# 14 | 0d 10h 46m 16s | 1d 12h 13m 12s
# 15 | 0d 14h 10m 40s | 2d 02h 23m 52s
# 16 | 0d 18h 19m 19s | 2d 20h 43m 11s
# 17 | 0d 23h 12m 16s | 3d 19h 55m 27s
# 18 | 1d 05h 12m 42s | 5d 01h 08m 09s
# 19 | 1d 12h 18m 36s | 6d 13h 26m 45s
# 20 | 1d 20h 33m 34s | 8d 10h 00m 19s
# 21 | 2d 06h 04m 10s | 10d 16h 04m 29s
# 22 | 2d 17h 11m 02s | 13d 09h 15m 31s
# 23 | 3d 05h 52m 40s | 16d 15h 08m 11s
# 24 | 3d 20h 21m 06s | 20d 11h 29m 17s
# 25 | 4d 12h 39m 20s | 25d 00h 08m 37s
def sidekiq_exponential_backoff(attempt)
(attempt**4) + 15 + (rand(30) * (attempt + 1))
end
# Builtin default Delayed::Job retry strategy.
#
# @see {Delayed::Backend::Base#reschedule_at}
#
# Approximate waiting times for given attempt for the builtin Delayed::Job retry logic.
#
# # | Next retry backoff | Total waiting time
# --------------------------------------------
# 0 | 0d 00h 00m 05s | 0d 00h 00m 05s
# 1 | 0d 00h 00m 06s | 0d 00h 00m 11s
# 2 | 0d 00h 00m 21s | 0d 00h 00m 32s
# 3 | 0d 00h 01m 26s | 0d 00h 01m 58s
# 4 | 0d 00h 04m 21s | 0d 00h 06m 19s
# 5 | 0d 00h 10m 30s | 0d 00h 16m 49s
# 6 | 0d 00h 21m 41s | 0d 00h 38m 30s
# 7 | 0d 00h 40m 06s | 0d 01h 18m 36s
# 8 | 0d 01h 08m 21s | 0d 02h 26m 57s
# 9 | 0d 01h 49m 26s | 0d 04h 16m 23s
# 10 | 0d 02h 46m 45s | 0d 07h 03m 08s
# 11 | 0d 04h 04m 06s | 0d 11h 07m 14s
# 12 | 0d 05h 45m 41s | 0d 16h 52m 55s
# 13 | 0d 07h 56m 06s | 1d 00h 49m 01s
# 14 | 0d 10h 40m 21s | 1d 11h 29m 22s
# 15 | 0d 14h 03m 50s | 2d 01h 33m 12s
# 16 | 0d 18h 12m 21s | 2d 19h 45m 33s
# 17 | 0d 23h 12m 06s | 3d 18h 57m 39s
# 18 | 1d 05h 09m 41s | 5d 00h 07m 20s
# 19 | 1d 12h 12m 06s | 6d 12h 19m 26s
# 20 | 1d 20h 26m 45s | 8d 08h 46m 11s
# 21 | 2d 06h 01m 26s | 10d 14h 47m 37s
# 22 | 2d 17h 04m 21s | 13d 07h 51m 58s
# 23 | 3d 05h 44m 06s | 16d 13h 36m 04s
# 24 | 3d 20h 09m 41s | 20d 09h 45m 45s
# 25 | 4d 12h 30m 30s | 24d 22h 16m 15s
def delayed_job_exponential_backoff(attempt)
(attempt**4) + 5
end
# Generic exponential retry strategy based on Sidekiq and Delayed::Job.
#
# Approximate waiting times for given attempt and default values:
#
# # | Next retry backoff | Total waiting time
# --------------------------------------------
# 0 | 0d 00h 00m 15s | 0d 00h 00m 15s
# 1 | 0d 00h 00m 16s | 0d 00h 00m 31s
# 2 | 0d 00h 00m 31s | 0d 00h 01m 02s
# 3 | 0d 00h 01m 36s | 0d 00h 02m 38s
# 4 | 0d 00h 04m 31s | 0d 00h 07m 09s
# 5 | 0d 00h 10m 40s | 0d 00h 17m 49s
# 6 | 0d 00h 21m 51s | 0d 00h 39m 40s
# 7 | 0d 00h 40m 16s | 0d 01h 19m 56s
# 8 | 0d 01h 08m 31s | 0d 02h 28m 27s
# 9 | 0d 01h 49m 36s | 0d 04h 18m 03s
# 10 | 0d 02h 46m 55s | 0d 07h 04m 58s
# 11 | 0d 04h 04m 16s | 0d 11h 09m 14s
# 12 | 0d 05h 45m 51s | 0d 16h 55m 05s
# 13 | 0d 07h 56m 16s | 1d 00h 51m 21s
# 14 | 0d 10h 40m 31s | 1d 11h 31m 52s
# 15 | 0d 14h 04m 00s | 2d 01h 35m 52s
# 16 | 0d 18h 12m 31s | 2d 19h 48m 23s
# 17 | 0d 23h 12m 16s | 3d 19h 00m 39s
# 18 | 1d 05h 09m 51s | 5d 00h 10m 30s
# 19 | 1d 12h 12m 16s | 6d 12h 22m 46s
# 20 | 1d 20h 26m 55s | 8d 08h 49m 41s
# 21 | 2d 06h 01m 36s | 10d 14h 51m 17s
# 22 | 2d 17h 04m 31s | 13d 07h 55m 48s
# 23 | 3d 05h 44m 16s | 16d 13h 40m 04s
# 24 | 3d 20h 09m 51s | 20d 09h 49m 55s
# 25 | 4d 12h 30m 40s | 24d 22h 20m 35s
def exponential_backoff(attempt, base: 0, exp: 4, delay: 15)
delay + (base + attempt)**exp
end
# Generic randomized exponential retry strategy based on Sidekiq and Delayed::Job.
#
# Approximate waiting times for given attempt and default values:
#
# | Next retry backoff | Total waiting time
# --------------------------------------------
# 0 | 0d 00h 00m 26s | 0d 00h 00m 26s
# 1 | 0d 00h 00m 38s | 0d 00h 01m 04s
# 2 | 0d 00h 00m 52s | 0d 00h 01m 56s
# 3 | 0d 00h 01m 52s | 0d 00h 03m 48s
# 4 | 0d 00h 06m 31s | 0d 00h 10m 19s
# 5 | 0d 00h 12m 16s | 0d 00h 22m 35s
# 6 | 0d 00h 21m 51s | 0d 00h 44m 26s
# 7 | 0d 00h 44m 00s | 0d 01h 28m 26s
# 8 | 0d 01h 10m 10s | 0d 02h 38m 36s
# 9 | 0d 01h 53m 56s | 0d 04h 32m 32s
# 10 | 0d 02h 50m 57s | 0d 07h 23m 29s
# 11 | 0d 04h 08m 16s | 0d 11h 31m 45s
# 12 | 0d 05h 49m 58s | 0d 17h 21m 43s
# 13 | 0d 07h 58m 08s | 1d 01h 19m 51s
# 14 | 0d 10h 41m 16s | 1d 12h 01m 07s
# 15 | 0d 14h 04m 32s | 2d 02h 05m 39s
# 16 | 0d 18h 15m 04s | 2d 20h 20m 43s
# 17 | 0d 23h 20m 40s | 3d 19h 41m 23s
# 18 | 1d 05h 13m 39s | 5d 00h 55m 02s
# 19 | 1d 12h 18m 56s | 6d 13h 13m 58s
# 20 | 1d 20h 32m 10s | 8d 09h 46m 08s
# 21 | 2d 06h 11m 30s | 10d 15h 57m 38s
# 22 | 2d 17h 11m 48s | 13d 09h 09m 26s
# 23 | 3d 05h 46m 40s | 16d 14h 56m 06s
# 24 | 3d 20h 19m 51s | 20d 11h 15m 57s
# 25 | 4d 12h 35m 00s | 24d 23h 50m 57s
def randomized_exponential_backoff(attempt, base: 0, exp: 4, delay: 15)
delay + (base + attempt)**exp + randomize(attempt)
end
# Generic constant retry strategy.
#
# Approximate waiting times for given attempt and default values:
#
# | Next retry backoff | Total waiting time
# --------------------------------------------
# 0 | 0d 00h 00m 15s | 0d 00h 00m 15s
# 1 | 0d 00h 00m 15s | 0d 00h 00m 30s
# 2 | 0d 00h 00m 15s | 0d 00h 00m 45s
# 3 | 0d 00h 00m 15s | 0d 00h 01m 00s
# 4 | 0d 00h 00m 15s | 0d 00h 01m 15s
# 5 | 0d 00h 00m 15s | 0d 00h 01m 30s
# 6 | 0d 00h 00m 15s | 0d 00h 01m 45s
# 7 | 0d 00h 00m 15s | 0d 00h 02m 00s
# 8 | 0d 00h 00m 15s | 0d 00h 02m 15s
# 9 | 0d 00h 00m 15s | 0d 00h 02m 30s
def constant_backoff(_attempt, delay: 15)
delay
end
# Generic randomized constant retry strategy.
#
# Approximate waiting times for given attempt and default values:
#
# # | Next retry backoff | Total waiting time
# --------------------------------------------
# 0 | 0d 00h 00m 19s | 0d 00h 00m 19s
# 1 | 0d 00h 00m 33s | 0d 00h 00m 52s
# 2 | 0d 00h 00m 44s | 0d 00h 01m 36s
# 3 | 0d 00h 00m 27s | 0d 00h 02m 03s
# 4 | 0d 00h 00m 35s | 0d 00h 02m 38s
# 5 | 0d 00h 00m 33s | 0d 00h 03m 11s
# 6 | 0d 00h 00m 35s | 0d 00h 03m 46s
# 7 | 0d 00h 00m 19s | 0d 00h 04m 05s
# 8 | 0d 00h 00m 22s | 0d 00h 04m 27s
# 9 | 0d 00h 00m 30s | 0d 00h 04m 57s
def randomized_constant_backoff(_attempt, delay: 15)
delay + randomize(0)
end
# Generic linear retry strategy.
#
# Approximate waiting times for given attempt and default values:
#
# # | Next retry backoff | Total waiting time
# --------------------------------------------
# 0 | 0d 00h 00m 15s | 0d 00h 00m 15s
# 1 | 0d 00h 01m 15s | 0d 00h 01m 30s
# 2 | 0d 00h 02m 15s | 0d 00h 03m 45s
# 3 | 0d 00h 03m 15s | 0d 00h 07m 00s
# 4 | 0d 00h 04m 15s | 0d 00h 11m 15s
# 5 | 0d 00h 05m 15s | 0d 00h 16m 30s
# 6 | 0d 00h 06m 15s | 0d 00h 22m 45s
# 7 | 0d 00h 07m 15s | 0d 00h 30m 00s
# 8 | 0d 00h 08m 15s | 0d 00h 38m 15s
# 9 | 0d 00h 09m 15s | 0d 00h 47m 30s
def linear_backoff(attempt, coefficient: 60, delay: 15)
delay + attempt * coefficient
end
# Generic randomized linear retry strategy.
#
# Approximate waiting times for given attempt and default values:
#
# # | Next retry backoff | Total waiting time
# --------------------------------------------
# 0 | 0d 00h 00m 26s | 0d 00h 00m 26s
# 1 | 0d 00h 02m 01s | 0d 00h 02m 27s
# 2 | 0d 00h 03m 18s | 0d 00h 05m 45s
# 3 | 0d 00h 04m 15s | 0d 00h 10m 00s
# 4 | 0d 00h 04m 15s | 0d 00h 14m 15s
# 5 | 0d 00h 06m 57s | 0d 00h 21m 12s
# 6 | 0d 00h 08m 14s | 0d 00h 29m 26s
# 7 | 0d 00h 11m 07s | 0d 00h 40m 33s
# 8 | 0d 00h 08m 42s | 0d 00h 49m 15s
# 9 | 0d 00h 12m 45s | 0d 01h 02m 00s
def randomized_linear_backoff(attempt, coefficient: 60, delay: 15)
delay + attempt * coefficient + randomize(attempt)
end
# Helper method to give a randomize effect to any retry strategy.
#
# Based on Sidekiq default retry logic:
# https://github.com/mperham/sidekiq/wiki/Error-Handling#automatic-job-retry
# https://github.com/mperham/sidekiq/issues/480
#
# Maximum waiting times for given attempt when rand = 30:
#
# # | Next retry backoff | Total waiting time
# --------------------------------------------
# 0 | 0d 00h 00m 30s | 0d 00h 00m 30s
# 1 | 0d 00h 01m 00s | 0d 00h 01m 30s
# 2 | 0d 00h 01m 30s | 0d 00h 03m 00s
# 3 | 0d 00h 02m 00s | 0d 00h 05m 00s
# 4 | 0d 00h 02m 30s | 0d 00h 07m 30s
# 5 | 0d 00h 03m 00s | 0d 00h 10m 30s
# 6 | 0d 00h 03m 30s | 0d 00h 14m 00s
# 7 | 0d 00h 04m 00s | 0d 00h 18m 00s
# 8 | 0d 00h 04m 30s | 0d 00h 22m 30s
# 9 | 0d 00h 05m 00s | 0d 00h 27m 30s
def randomize(attempt)
rand(30) * (attempt + 1)
end
# Renders table of waiting times for various rescheduling strategies.
#
# Inspiration taken from https://github.com/mperham/sidekiq/wiki/Error-Handling#automatic-job-retry.
#
# @param [Integer] attempts the maximum number of attempts
# @param [Integer] width the column width
#
# @return [String] the rendered table
#
# @example Usage
# puts JobRetry.retry_table(attempts: 25) { |attempt| JobsRetry.linear_backoff(attempt) }
#
def retry_table(attempts: 25, col_width: 18)
out = " # | %#{col_width}s | %#{col_width}s\n" % ['Next retry backoff', 'Total waiting time']
out += "#{'-' * (out.size - 1)}\n"
total_wait_time = 0
(0..attempts).each do |attempt|
wait_time = yield(attempt)
total_wait_time += wait_time
out += "%2d | %#{col_width}s | %#{col_width}s\n" % [
attempt,
humanize_duration(wait_time),
humanize_duration(total_wait_time)
]
end
out
end
# Utility method to render time duration as human readable string.
#
# @param [Integer] duration the time duration in seconds
#
# @return [String] duration as string formatted as "0d 00h 00m 00s"
#
def humanize_duration(duration)
units = %i[day hour minute second]
out = {}
units.reduce(duration.to_i) do |dur, unit|
out[unit], remaining_dur = dur.divmod(1.public_send(unit))
remaining_dur
end
'%<day>dd %<hour>02dh %<minute>02dm %<second>02ds' % out
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment