Skip to content

Instantly share code, notes, and snippets.

View mvillarrealb's full-sized avatar
🏠
Working from home

Marco Villarreal mvillarrealb

🏠
Working from home
View GitHub Profile
@mvillarrealb
mvillarrealb / spark-submit.sh
Last active September 23, 2018 17:29
A custom docker image based on spark-base:2.3.1 image, used to create a spark-submit containers that runs and die once they delivered the driver program to the cluster
#!/bin/bash
/spark/bin/spark-submit \
--class ${SPARK_APPLICATION_MAIN_CLASS} \
--master ${SPARK_MASTER_URL} \
--deploy-mode cluster \
--total-executor-cores 1 \
${SPARK_SUBMIT_ARGS} \
${SPARK_APPLICATION_JAR_LOCATION} \
${SPARK_APPLICATION_ARGS}
@mvillarrealb
mvillarrealb / spark-worker:2.3.1.Dockerfile
Last active September 23, 2018 17:26
A custom docker image based on spark-base:2.3.1 image, used to create a spark worker containers
FROM spark-base:2.3.1
COPY start-worker.sh /
ENV SPARK_WORKER_WEBUI_PORT 8081
ENV SPARK_WORKER_LOG /spark/logs
ENV SPARK_MASTER "spark://spark-master:7077"
EXPOSE 8081
@mvillarrealb
mvillarrealb / spark-master:2.3.1.Dockerfile
Last active November 10, 2019 14:31
A custom docker image based on spark-base:2.3.1 image, used to create a spark master containers
FROM spark-base:2.3.1
COPY start-master.sh /
ENV SPARK_MASTER_PORT 7077
ENV SPARK_MASTER_WEBUI_PORT 8080
ENV SPARK_MASTER_LOG /spark/logs
EXPOSE 8080 7077 6066
@mvillarrealb
mvillarrealb / spark-base:2.3.1.Dockerfile
Last active September 23, 2018 17:26
A base image based on java:alpine-jdk-8 wich ships scala, python3 and spark 2.3.1
FROM java:8-jdk-alpine
ENV DAEMON_RUN=true
ENV SPARK_VERSION=2.3.1
ENV HADOOP_VERSION=2.7
ENV SCALA_VERSION=2.12.4
ENV SCALA_HOME=/usr/share/scala
RUN apk add --no-cache --virtual=.build-dependencies wget ca-certificates && \
apk add --no-cache bash curl jq && \