Skip to content

Instantly share code, notes, and snippets.

@ikouchiha47
Last active September 19, 2016 14:04
Show Gist options
  • Save ikouchiha47/ba89b2a06c35069b9003df0d94e06725 to your computer and use it in GitHub Desktop.
Save ikouchiha47/ba89b2a06c35069b9003df0d94e06725 to your computer and use it in GitHub Desktop.
ruby csv parser memory test
class Config
class << self
def get
env = ENV["RUBY_ENV"] || "developement"
env = env.downcase
require_relative "./environments/" + downcase
end
end
end
require_relative './helpers'
require 'csv'
print_memory_usage do
print_time_spent do
csv = CSV.open('./data/test_data.csv')
csv.drop(13).each do |row|
p row
end
CSV.foreach('./data/test_data.csv') do |row|
p row
end
end
end
require 'benchmark'
def print_memory_usage
memory_before = `ps -o rss= -p #{Process.pid}`.to_i
yield
memory_after = `ps -o rss= -p #{Process.pid}`.to_i
puts "Memory: #{((memory_after - memory_before) / 1024.0).round(2)} MB"
end
def print_time_spent
time = Benchmark.realtime do
yield
end
puts "Time: #{time.round(2)}"
end
We can make this file beautiful and searchable if this error is corrected: It looks like row 13 should actually have 2 columns, instead of 12 in line 12.
Report,Reviews
App ID,com.gojek.app
App,GO-JEK
Version,All Versions
Market,Google Play
Language,All Languages
Total Reviews,6603
Average Rating,3.5
Search Term,
Start Date,08/13/2016
End Date,09/12/2016
Date Generated,09/14/2016 00:50:51 PDT
Platform,Language,Date,App ID,App Name,Publisher ID,Publisher Name,User,Version,Rating,Title,Review
Google Play,English,09/12/2016,com.gojek.app,GO-JEK,20200000891258,PT GO-JEK INDONESIA,,,1,," Saldo saya hilang, padahal saya sebelumnya top up 2x dan belum terpakai "
Google Play,English,09/12/2016,com.gojek.app,GO-JEK,20200000891258,PT GO-JEK INDONESIA,,,4,Rew, Dwwww
Google Play,English,09/12/2016,com.gojek.app,GO-JEK,20200000891258,PT GO-JEK INDONESIA,,,2,A little bit disappointed," Cari driver lsg ""booking in progress"", tp ditunggu 15menit gk dtg. Ditelp gk diangkat. Cancel booking trus find lg, ditlp gk diangkat jg pdhl beda driver. Cukup. "
Google Play,English,09/12/2016,com.gojek.app,GO-JEK,20200000891258,PT GO-JEK INDONESIA,,,5,Smart, Very Easy take something
@matthewd
Copy link

File.open('./data/test_data.csv') do |io|
  nil until io.gets == "\n"

  csv = CSV.new(io, headers: true)
  csv.each do |row|
    p row
  end
end

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment