Skip to content

Instantly share code, notes, and snippets.

@evidanary
Created February 24, 2014 22:27
Show Gist options
  • Save evidanary/9198578 to your computer and use it in GitHub Desktop.
Save evidanary/9198578 to your computer and use it in GitHub Desktop.
This gets mixpanel JSON data and uploads to a table in HIVE
# Download Mixpanel Data
require '../lib/hive_utils.rb'
require 'yaml'
require 'fileutils'
require File.expand_path(File.dirname(__FILE__) + '/mixpanel_extract.rb')
# TODO use thor here to take the date from the commandline
def print_usage_and_exit
puts %q[Usage:
ruby mixpanel_download_events.rb <date in YYYY-MM-DD> <config_file> [show_uri]
e.g. ruby mixpane_download_dashboard_events.rb 2014-01-20 /var/lib/jenkins/config/mixpanel_dashboard.yml
The show_uri optional argument will show the uri to download the file and will NOT process the file]
exit 1
end
#config, start_date = nil, nil
print_usage_and_exit if ARGV.length < 2
begin
Date.parse(ARGV[0])
start_date = ARGV[0]
config = YAML.load_file(ARGV[1])
rescue
print_usage_and_exit
end
# Download File
mixpanel = MixpanelExtract.new(config, start_date)
# show uri
if ARGV[2] == 'show_uri'
config[:limit] = 5 # get sample json
puts "Mixpanel Download URI: "
puts mixpanel.request_uri
exit 0
end
mixpanel.process_date
puts "Done processing Download for #{start_date}"
# Convert JSON to CSV
file_in = mixpanel.filename
file_out = file_in.gsub(/json$/, 'csv')
mixpanel.transform(file_in, file_out)
puts "Done transforming #{file_in} to #{file_out}"
#Upload to Hadoop
DataUtils::Hive.load_table_partition(file_out,
config[:part_field],
start_date,
config[:hive_schema],
config[:hive_table])
# FileUtils.rm(file_in)
# FileUtils.rm(file_out)
puts %Q[Done Loading File #{file_out} to #{config[:hive_schema]}.#{config[:hive_table]} \
with partition #{config[:part_field]} = #{start_date}]
@derwin12
Copy link

derwin12 commented Apr 1, 2014

Are you able to post a sample config file?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment