Ivan Leichtling ivanlei

ivanlei / mrjob_pooling_with_multiple_iam.sh

Created July 20, 2013 16:30

Example command for running an mrjob with job pooling and EMR instances shared between IAM in the same AWS account.

	export AWS_ACCESS_KEY_ID=XXX
	export AWS_SECRET_ACCESS_KEY=XXX
	python ./mr_word_freq_count.py ./wordlist \
	--pool-emr-job-flows \
	--runner=emr \
	--visible-to-all-users \
	--num-ec2-instances=1 \
	--aws-region=us-west-2 --emr-endpoint=us-west-2.elasticmapreduce.amazonaws.com

ivanlei / mrjob_install.sh

Created July 20, 2013 16:24

Install the python package mrjob from source

	sudo apt-get update
	sudo apt-get install -y git-core python-pip python-dev build-essential
	sudo pip install --upgrade pip
	sudo pip install --upgrade virtualenv
	sudo pip install boto
	sudo pip install simplejson
	sudo git clone https://github.com/Yelp/mrjob.git
	pushd mrjob
	sudo python setup.py install
	popd