Skip to content

Instantly share code, notes, and snippets.

@saswata-dutta
Forked from yaravind/spark-rest-submit.sh
Created May 31, 2019 07:16
Show Gist options
  • Save saswata-dutta/91341c31f24cb24cd987cf4a0e9b36f2 to your computer and use it in GitHub Desktop.
Save saswata-dutta/91341c31f24cb24cd987cf4a0e9b36f2 to your computer and use it in GitHub Desktop.
Submit apps (SparkPi as e.g.) to spark cluster using rest api
curl -X POST -d http://master-host:6066/v1/submissions/create --header "Content-Type:application/json" --data '{
"action": "CreateSubmissionRequest",
"appResource": "hdfs://localhost:9000/user/spark-examples_2.11-2.0.0.jar",
"clientSparkVersion": "2.0.0",
"appArgs": [ "10" ],
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass": "org.apache.spark.examples.SparkPi",
"sparkProperties": {
"spark.jars": "hdfs://localhost:9000/user/spark-examples_2.11-2.0.0.jar",
"spark.driver.supervise":"false",
"spark.executor.memory": "512m",
"spark.driver.memory": "512m",
"spark.submit.deployMode":"cluster",
"spark.app.name": "SparkPi",
"spark.master": "spark://master-host:6066"
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment