Skip to content

Instantly share code, notes, and snippets.

@jmtame
Last active August 29, 2015 14:11
Show Gist options
  • Save jmtame/8aa3e817d9f4445d377f to your computer and use it in GitHub Desktop.
Save jmtame/8aa3e817d9f4445d377f to your computer and use it in GitHub Desktop.
batch parallel processing
jobs_per_process = 100
process_count = 20
HolidayCampaignBidLapse.where('user_id % 2 = 0').where(sent: false).find_in_batches(batch_size: jobs_per_process * process_count) do |group|
batches = group.in_groups(process_count)
batches.each do |batch|
Process.fork do
ActiveRecord::Base.establish_connection
batch.each do |campaign|
credit = HolidayCreditBidLapse.find_or_create_by_user_id(campaign.user_id, :amount => 5)
campaign.update_column :sent, true
end
end
end
Process.waitall
return if HolidayCreditBidLapse.count >= 300000
end
# Watch send rate
loop do; sleep 1.minute; puts HolidayCreditBidLapse.count end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment