Madhup Sukoon madhup

madhup / spark_worker_google_colab.py

Created July 4, 2021 16:27

Spark Worker on Google Colab

	!npm install -g localtunnel
	!lt --port 8081 > localtunnel.log 2>&1 &
	!apt-get install openjdk-8-jdk-headless -qq > /dev/null
	!wget -q https://downloads.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
	!tar -xvf spark-3.1.2-bin-hadoop3.2.tgz > /dev/null
	!cat localtunnel.log
	!tail -f "$(/content/spark-3.1.2-bin-hadoop3.2/sbin/start-slave.sh --memory 10G spark://xx.xx.xx.xx:7077 \| cut -d ' ' -f 5)"