Skip to content

Instantly share code, notes, and snippets.

@bcackerman
Created August 17, 2016 20:51
Show Gist options
  • Save bcackerman/1d1adb7db4116a1d776c4e8173c12abb to your computer and use it in GitHub Desktop.
Save bcackerman/1d1adb7db4116a1d776c4e8173c12abb to your computer and use it in GitHub Desktop.
Code to download a CSV file on Heroku (using Rails) and run a script on it
# I'm using smarter-csv gem to help parse the CSV
require 'net/ftp'
filename = 'filename-here.csv'
temp_file = Tempfile.new("download-#{filename}")
temp_file.binmode
import_options = {chunk_size: 10000, downcase_header: true, :row_sep => :auto, encoding: "UTF-8"}
size = 0
progress = 0
Net::FTP.open('ftp.domain.com') do |ftp|
ftp.passive = true
ftp.login 'username', 'password'
ftp.chdir('/direct-if-there-is-one')
total = ftp.size(filename)
ftp.getbinaryfile(filename, nil, 8192) do |chunk|
temp_file << chunk
size += chunk.size
new_progress = (size * 100) / total
unless new_progress == progress
puts "\rDownloading %s (%3d%%) " % [filename, new_progress]
end
progress = new_progress
end
end
SmarterCSV.process(temp_file, import_options) do |chunk|
chunk.each do |row|
puts row
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment