Skip to content

Instantly share code, notes, and snippets.

@we4tech
Last active December 31, 2015 11:59
Show Gist options
  • Select an option

  • Save we4tech/7982974 to your computer and use it in GitHub Desktop.

Select an option

Save we4tech/7982974 to your computer and use it in GitHub Desktop.
An illustration how we can use ruby thread for accelerating our normal RAKE tasks. (intended for posting in a local Ruby developers group https://www.facebook.com/groups/rubydevs/)
namespace :somestuffs do
desc 'Importing some big stuff that requies lotta network stuff'
task :import
file, tasks = ENV['FILE'], Queue.new
# Parse out CSV file and retrieve each row and queue them up
# for further processing.
#
# Keeping it a thread allows us to let this process continue while other
# threads are not in RUNNING state.
Thread.new do
CSV.open(file) { |row| tasks << row }
end
# Use something that matters in your envrionment,
# if you are using some network call which takes quite long time,
# then you can use 10/20 threads / CPU core
4.times do
Thread.new do
while (row = tasks.pop)
# Do network transfer related blocking task.
end
end
end
# Control thread which keeps current process blocked until queue becomes zero
# In our case we are expecting tasks.size could get zero whenever no more data to process.
Thread.new do
sleep 1 until tasks.size.zero?
end.join
# Calling "join" allows current process to wait until the JOINED thread is finished.
# Thus we can ensure our parent process doesn't exit until all tasks are done.
end
end
@we4tech
Copy link
Author

we4tech commented Dec 19, 2013

thanks @mftaher bhai :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment