Pyspark Kernel in Anaconda

Custom_Pyspark_kernel

### Find the kernel.json location 
jupyter kernelspec list

###Create Pyspark Folder and create Kernel.json file  with below content
{
  "display_name": "pyspark",
  "language": "python",
  "argv": [ "/opt/anaconda3/bin/python", "-m", "ipykernel", "-f", "{connection_file}" ],
  "env": {
    "JAVA_HOME": "/usr/lib/jvm/java-8-openjdk-amd64",
    "SPARK_HOME": "/usr/lib/spark",
    "PYSPARK_PYTHON": "/opt/anaconda3/bin/python",
    "PYTHONSTARTUP": "/usr/lib/spark/python/pyspark/shell.py",
    "PYTHONPATH": "/usr/lib/spark/python/:/usr/lib/spark/python/lib/py4j-0.10.7-src.zip",
    "PYSPARK_SUBMIT_ARGS": "--master yarn --deploy-mode client pyspark-shell"
    
  }
}