Describe how to submit Spark application into a cluster and showing the result inside Intellij IDEA

run_spark_inside_intellij_idea.md

1. Add bin folder of your Spark installation into the $PATH environmental variable

2. Add a SBT Task configuration and input package inside the Tasks box

3. Uncheck the Activate tool window and press Apply

4. Add a Shell Script configuration and select Script text as Execute: value

5. Insert

spark-submit --class "<Your Class Package>.MainClass" --packages <REQUIRED PACKAGE TO RUN COMMA SEPARATED> --master "spark://<CLUSTER DOMAIN>:7077" target/scala-<SCALA VERSION>/<PROJECT NAME>_<SCALA VERSION>-<PROJECT VERSION>.jar

As Script text and check the Execute in terminal option. Fill <VALUES> at previous command with appropriate values.

6. On Before launch section add the SBT Task created on step 2.

7. Press OK.

8. Run application from Select Run Configuration Box on the top of the main window.

An example of MainClass on the previous command:

package ir.mpgutils.spark.test

import org.apache.spark.sql.SparkSession
import org.apache.spark.SparkConf

object MyMainClass {
    def main(args: Array[String]) {
        val conf = new SparkConf()
            .setAppName("Test Run From Intellij IDEA")

        val spark = SparkSession.builder.config(conf).getOrCreate()

        spark.stop()
    }
}

An example of command to run:

spark-submit --class "ir.mpgutils.spark.test.MyMainClass"  --master "spark://freedom:7077" target/scala-2.12/test_run_from_intellij_2.12-0.1.jar