Andrea Nicolai andrybicio
rao-abdul-mannan
/ Spark2.2_CDH5.13_Jupyterhub
Last active
March 30, 2021 07:28
This gist explains how to connect jupyterhub with Spark2 on CDH 5.13 Cluster
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ### This gist explains how to connect jupyterhub with Spark2 on CDH 5.13 Cluster. | |
| Following below instructions, Spark can be configured with Jupyterhub on any cluster, standalone or locally etc | |
| - Install jupyterhub by following instructions on official repo https://github.com/jupyterhub/jupyterhub | |
| - Once installed, before configuring spark2 kernel first locate the jupyter's kernels directory. | |
| On centos7, its under /usr/share/jupyter/kernels/ | |
| - Assuming all jupyterhub kernel's are in /usr/share/jupyter/kernels/ | |
| mkdir /usr/share/jupyter/kernels/pyspark2 | |
| - Create pyspark2 kernel | |
| touch /usr/share/jupyter/kernels/pyspark2/kernel.json |