Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save dennisfaust/c94f0e1aec54e37c52e431e9542ff042 to your computer and use it in GitHub Desktop.
Save dennisfaust/c94f0e1aec54e37c52e431e9542ff042 to your computer and use it in GitHub Desktop.
sidekiq-unique-jobs gem not deleting expired keys in its uniquejobs hash
# https://github.com/mhenrixon/sidekiq-unique-jobs/issues/161
# Even worse: https://github.com/mhenrixon/sidekiq-unique-jobs/issues/234
class SidekiqUniqueJobsHashCompactor
include Sidekiq::Worker
sidekiq_options queue: "slow"
def perform
# Skip if there are jobs queued...
return unless Sidekiq::Queue.all.select { |q| q.size > 100 }.blank?
# We need this to get a redis, db12 with no namespace gem
redis = Redis.new(host: $redis.client.host, port: $redis.client.port, db: 12, thread_safe: true)
to_be_deleted = []
offset_str = redis.get("uniquejobscursor")
offset = offset_str.try(:to_i) || 0
cursor, jobs = redis.hscan("uniquejobs", offset, count: 2_000)
redis.set("uniquejobscursor", cursor)
jobs.each do |job_array|
jid, unique_key = job_array
next if redis.get(unique_key)
to_be_deleted << jid
end
to_be_deleted.blank? ? num_del = 0 : num_del = redis.hdel("uniquejobs", to_be_deleted)
FDStats.count('uniquejobs_hash_compactor', num_del)
end
end
# start = Time.now; SidekiqUniqueJobsHashCompactor.new.perform; puts "#{(Time.now - start) * 1000} milliseconds"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment