Skip to content

Instantly share code, notes, and snippets.

@koduki

koduki/dataproc.md

Last active October 21, 2016 16:56
Show Gist options
  • Save koduki/3818e1cef0dfb4aefd93b57ff635aeac to your computer and use it in GitHub Desktop.
Save koduki/3818e1cef0dfb4aefd93b57ff635aeac to your computer and use it in GitHub Desktop.
Download ZIP
Raw
dataproc.md

setup cluster

$ gcloud dataproc clusters create --master-machine-type "n5-standard-4" --worker-machine-type "n1-standard-4" --num-preemptible-workers 1 mycluster

spark-submit

$ gcloud dataproc jobs submit spark --properties spark.dynamicAllocation.enabled=false --cluster mycluster --class techconf.example.spark01.Main --jars Documents\NetBeansProjects\example-spark01\target\example-spark01-0.1.jar

web ui.

$ gcloud compute ssh --zone=us-central1-f --ssh-flag="-D" --ssh-flag="1080" --ssh-flag="-N" mycluster-m
$ opne http://mycluster-m:8088/ # with proxy 1080

cleanup cluster

$ gcloud dataproc clusters delete mycluster
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment