Skip to content

Instantly share code, notes, and snippets.

@theotherzach
Created April 12, 2013 14:05
Show Gist options
  • Select an option

  • Save theotherzach/5372250 to your computer and use it in GitHub Desktop.

Select an option

Save theotherzach/5372250 to your computer and use it in GitHub Desktop.
strategy for splitting hit file efficiently
require 'set'
require 'csv'
data_path = '/Users/zach/Code/metrics_loader/data/'
$hits_path = "#{data_path}/analytic_recording_tags_20121216.csv"
$recordings_path = "#{data_path}/recordings_20121216.csv"
$raw_ids = Set.new
IO.foreach($hits_path).each do |line|
next if(ids = line.slice!(0, 13)) == 'record_ident|'
$raw_ids.add ids
end
$hit_ids = Set.new
$raw_ids.each do |id|
$hit_ids.add({
recording_id: Integer(id.slice(/^\d+(?=\|)/)),
tag_id: Integer(id.slice(/(?<=\|)\d+/))
})
end
$raw_ids.clear
$recordings = []
class Recording
attr_accessor :id, :recording_time, :inbound, :tags
end
CSV.foreach($recordings_path, headers: true, col_sep: '|') do |line|
recording = Recording.new
recording.id = line.fetch('ident')
recording.recording_time = line.fetch('recording_time')
$recordings << recording
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment