Skip to content

Instantly share code, notes, and snippets.

@bigsur0
Created September 23, 2011 16:53
Show Gist options
  • Save bigsur0/1237857 to your computer and use it in GitHub Desktop.
Save bigsur0/1237857 to your computer and use it in GitHub Desktop.
Parallel gem and find_in_batches with start, end
num_processes = 2
batch_size = 1000
threads_per_process = 10
Parallel.map(0...num_processes, :in_processes => num_processes) do |i|·
User.connection.reconnect!
User.find_in_batches(:batch_size => batch_size,·
:start => (User.count/num_processes) * i,·
:conditions => ["id <= ?", (User.count/num_processes)*(i+1)]) do |batch|·
Parallel.map(batch, :in_threads => threads_per_process) do |record|
puts record.inspect
end·
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment