Skip to content

Instantly share code, notes, and snippets.

@thattommyhall
Created May 31, 2011 16:51
Show Gist options
  • Select an option

  • Save thattommyhall/1000863 to your computer and use it in GitHub Desktop.

Select an option

Save thattommyhall/1000863 to your computer and use it in GitHub Desktop.
First attempt, loop through partitions using rbhive
require 'rubygems'
require 'date'
require 'rbhive'
countrys = %w[at au br de dk es fr in it jp mx nl no pl pt ru se uk us za]
dates = (Date.parse('2011-01-01')..Date.parse('2011-04-30'))
RBHive.connect('hiveserver') do |con|
dates.each do |date|
countrys.each do |country|
query = "insert overwrite table keywords partition (dated='#{date}', country = '#{country}')
select account,campaign,ad_group,keyword_id,keyword,match_type,status,
first_page_bid,quality_score,distribution,max_cpc,destination_url,ad_group_status,
campaign_status,currency_code,impressions,clicks,ctr,cpc,
cost,avg_position,account_id,campaign_id,adgroup_id
from keywords where dated='#{date}' and country='#{country}'"
begin
con.set('mapred.output.compression.codec','org.apache.hadoop.io.compress.GzipCodec')
con.set('hive.exec.compress.output','true')
con.set('mapred.output.compress','true')
con.set('mapred.compress.map.output','true')
con.set('hive.merge.mapredfiles','true')
con.set('hive.merge.mapfiles','true')
con.execute(query)
rescue => e
puts "#########################"
puts e.message
puts "#########################"
end
end
end
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment