Created
April 5, 2017 19:58
-
-
Save jagthedrummer/9b18b9c29a489357fb7ee1634bc2049d to your computer and use it in GitHub Desktop.
Rake task to transfer Sidekiq jobs from one redis instance to another
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# This task should be run inside an environment that is already configured to connect to the redis | |
# instance that we're transfering AWAY FROM. | |
# | |
# The task should be handed the URL of the redis instance that we're MOVING TO. | |
# | |
# To run it and pass in the destination Redis you'd do something like this: | |
# rake sidekiq:transfer[redis://...] | |
# | |
# As jobs are added to the destination Redis, they're deleted from the source Redis. This | |
# allows the task to be restarted cleanly if it fails in the middle due to a network error | |
# or something like that. | |
# | |
namespace :sidekiq do | |
desc 'Transfer sidekiq jobs from one instance to another' | |
task :transfer, [:destination] => :environment do |task, args| | |
def transfer_queue queue, destination_client | |
puts "transfering queue : #{queue.name}" | |
queue.each do |job| | |
destination_client.push 'class' => job.klass, 'args' => job.args | |
job.delete | |
end | |
end | |
def transfer_set set, destination_client | |
set.each do |job| | |
destination_client.push 'class' => job.klass, 'args' => job.args, 'at' => job.at.to_i | |
scheduled_job.delete | |
end | |
end | |
destination_pool = ConnectionPool.new { Redis.new(url: args[:destination]) } | |
destination_client = Sidekiq::Client.new(destination_pool) | |
source_queues = Sidekiq::Queue.all | |
source_queues.each do |queue| | |
transfer_queue queue, destination_client | |
end | |
puts "transfering scheduled jobs" | |
ss = Sidekiq::ScheduledSet.new | |
transfer_set ss, destination_client | |
puts "transfering retried jobs" | |
rs = Sidekiq::RetrySet.new | |
transfer_set rs, destination_client | |
end | |
end |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanks a lot 🙌
This is my procedural version I've used today, which includes checks on queue size to validate the migration: