yu-iskw/gist:2f1329ed5b0922460e80

Last active August 29, 2015 14:24

Star (0) You must be signed in to star a gist
Fork (0) You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/yu-iskw/2f1329ed5b0922460e80.js"></script>
Save yu-iskw/2f1329ed5b0922460e80 to your computer and use it in GitHub Desktop.

A command to launch a Spark cluster on EC2

Raw

_REGION='ap-northeast-1'
_ZONE='ap-northeast-1b'
_VERSION='1.4.0'
_MASTER_INSTANCE_TYPE='r3.large'
_SLAVE_INSTANCE_TYPE='r3.8xlarge'
_SLAVES=5
_PRICE=1.0
_CLUSTER_NAME="spark-cluster-v${_VERSION}-${_SLAVE_INSTANCE_TYPE}x${_SLAVES}"

./spark-ec2 -k "yu_ishikawa@recruit" -i ~/.ec2/yu_ishikawarecruit.pem -s $_SLAVES --master-instance-type="$_MASTER_INSTANCE_TYPE" --instance-type="$_SLAVE_INSTANCE_TYPE" --region="$_REGION" --zone="$_ZONE" --spot-price=$_PRICE --spark-version="${_VERSION}" --hadoop-major-version=2 launch "$_CLUSTER_NAME"