Skip to content

Instantly share code, notes, and snippets.

@j3speaks

j3speaks/Dockerfile

Last active May 7, 2019 02:23
Show Gist options
  • Save j3speaks/08e47c5dfce787030d16a4970c49ac06 to your computer and use it in GitHub Desktop.
Save j3speaks/08e47c5dfce787030d16a4970c49ac06 to your computer and use it in GitHub Desktop.
Download ZIP
Spark in a Docker container
Raw
Dockerfile
FROM openjdk:8-alpine
RUN apk --update add wget tar bash
RUN wget http://apache.mirror.anlx.net/spark/spark-2.4.2/spark-2.4.2-bin-hadoop2.7.tgz
RUN tar -xzf spark-2.4.2-bin-hadoop2.7.tgz && mv spark-2.4.2-bin-hadoop2.7 spark && rm spark-2.4.2-bin-hadoop2.7.tgz
RUN printf "#!/bin/sh\n/spark/bin/spark-class org.apache.spark.deploy.master.Master --host \$SPARK_MASTER_HOST --port \$SPARK_MASTER_PORT --webui-port \$SPARK_MASTER_WEBUI_PORT" > /start-master.sh
RUN chmod +x /start-master.sh
RUN printf "#!/bin/sh\n/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port \$SPARK_WORKER_WEBUI_PORT \$SPARK_MASTER_URL" > /start-worker.sh
RUN chmod +x /start-worker.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment