Skip to content

Instantly share code, notes, and snippets.

@lichti
Created November 3, 2015 16:44
Show Gist options
  • Save lichti/b0e089e579887659f633 to your computer and use it in GitHub Desktop.
Save lichti/b0e089e579887659f633 to your computer and use it in GitHub Desktop.
MySQL LOAD DATA parallel import
require 'thread'
require 'fileutils'
procs=8
host=""
user=""
pass=""
db=""
path=""
work = Queue.new
Dir.glob(File.join(path,"*.txt")) {|x| work.push x }
workers = (0...procs).map do
Thread.new do
begin
while x = work.pop(true)
file=x.split('.')
table=file[0].split('/').last
system "echo #{file[0]}.txt ; mysql -h#{host} -u#{user} -p#{pass} -e \"use #{db}; LOAD DATA LOCAL INFILE '#{file[0]}.txt' INTO TABLE #{table} FIELDS TERMINATED BY ',' ENCLOSED BY '\\\"' LINES TERMINATED BY '\\r\\n'; \" "
end
rescue ThreadError
end
end
end
workers.map(&:join)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment