Skip to content

Instantly share code, notes, and snippets.

@ivanlei

ivanlei/mrjob_pooling_with_multiple_iam.sh

Created July 20, 2013 16:30
Show Gist options
  • Save ivanlei/6045631 to your computer and use it in GitHub Desktop.
Save ivanlei/6045631 to your computer and use it in GitHub Desktop.
Download ZIP
Example command for running an mrjob with job pooling and EMR instances shared between IAM in the same AWS account.
Raw
mrjob_pooling_with_multiple_iam.sh
export AWS_ACCESS_KEY_ID=XXX
export AWS_SECRET_ACCESS_KEY=XXX
python ./mr_word_freq_count.py ./wordlist \
--pool-emr-job-flows \
--runner=emr \
--visible-to-all-users \
--num-ec2-instances=1 \
--aws-region=us-west-2 --emr-endpoint=us-west-2.elasticmapreduce.amazonaws.com
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment