An example submit to Apache Livy. #livy #spark

README-submit-to-livy.md

README - Submit to Livy using REST API

See https://livy.apache.org/docs/latest/rest-api.html

Let app.json be the JSON payload that represents the application:

{
    "file": "hdfs:///user/user/apps/hello-spark-0.0.1.jar",
    "className": "acme.hello_spark.Grep",
    "args": ["input/1.txt", "[Mm]agic"],
    "jars": [
       "hdfs:///user/user/jars/log4j-api-2.7.jar",
       "hdfs:///user/user/jars/log4j-core-2.7.jar"
    ],
    "driverMemory": "1g", 
    "executorMemory": "1g",
    "executorCores": 2, 
    "numExecutors": 2
}

Submit the application:

curl -d @app.json -H 'Content-Type: application/json' -X POST http://hadoop.example.net:8998/batches

The submission endpoint supports doAs request parameter to allow proxying other users (corresponds to --proxy-user option of spark-submit):

curl -d @app.json -H 'Content-Type: application/json' -X POST http://hadoop.example.net:8998/batches?doAs=someone

Fetch the status of submitted batch jobs:

curl http://hadoop.example.net:8998/batches

Fetch logs for a specific job (eg the one identified as 17):

curl http://hadoop.example.net:8998/batches/17/log