Skip to content

Instantly share code, notes, and snippets.

@paulosuzart

paulosuzart/mr.py

Created February 23, 2012 01:15
Show Gist options
  • Save paulosuzart/1888969 to your computer and use it in GitHub Desktop.
Save paulosuzart/1888969 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
mr.py
# Boto simpy works. Fuck Yeah!
from boto.emr.connection import EmrConnection
from boto.emr.step import JarStep
conn = EmrConnection('<Your Key ID>', '<Your Key>')
froms3 = JarStep(name='cpfroms3', jar='s3://us-east-1.elasticmapreduce/libs/s3distcp/1.0/s3distcp.jar', step_args=['--src', 's3n://cascastro/','--dest', 'hdfs:///tmp/users/'])
ex = JarStep(name='exec', jar='s3n://bills-prd/astrocomp-0.0.1-standalone.jar', step_args=['compat.astro', '/tmp/users', '/tmp/result/users'])
tos3 = JarStep(name='cptos3', jar='s3://us-east-1.elasticmapreduce/libs/s3distcp/1.0/s3distcp.jar', step_args=['--src', 'hdfs:///tmp/result/users','--dest', 's3n://ectwotools/'])
jid = conn.run_jobflow(name='PQP21', log_uri='s3n://ectwotools/', steps=[froms3, ex, tos3], num_instances=4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment