Skip to content

Instantly share code, notes, and snippets.

View mvillarrealb's full-sized avatar
🏠
Working from home

Marco Villarreal mvillarrealb

🏠
Working from home
View GitHub Profile
@mvillarrealb
mvillarrealb / spark-base:2.3.1.Dockerfile
Last active September 23, 2018 17:26
A base image based on java:alpine-jdk-8 wich ships scala, python3 and spark 2.3.1
FROM java:8-jdk-alpine
ENV DAEMON_RUN=true
ENV SPARK_VERSION=2.3.1
ENV HADOOP_VERSION=2.7
ENV SCALA_VERSION=2.12.4
ENV SCALA_HOME=/usr/share/scala
RUN apk add --no-cache --virtual=.build-dependencies wget ca-certificates && \
apk add --no-cache bash curl jq && \
@mvillarrealb
mvillarrealb / spark-master:2.3.1.Dockerfile
Last active November 10, 2019 14:31
A custom docker image based on spark-base:2.3.1 image, used to create a spark master containers
FROM spark-base:2.3.1
COPY start-master.sh /
ENV SPARK_MASTER_PORT 7077
ENV SPARK_MASTER_WEBUI_PORT 8080
ENV SPARK_MASTER_LOG /spark/logs
EXPOSE 8080 7077 6066
@mvillarrealb
mvillarrealb / spark-worker:2.3.1.Dockerfile
Last active September 23, 2018 17:26
A custom docker image based on spark-base:2.3.1 image, used to create a spark worker containers
FROM spark-base:2.3.1
COPY start-worker.sh /
ENV SPARK_WORKER_WEBUI_PORT 8081
ENV SPARK_WORKER_LOG /spark/logs
ENV SPARK_MASTER "spark://spark-master:7077"
EXPOSE 8081
@mvillarrealb
mvillarrealb / spark-submit.sh
Last active September 23, 2018 17:29
A custom docker image based on spark-base:2.3.1 image, used to create a spark-submit containers that runs and die once they delivered the driver program to the cluster
#!/bin/bash
/spark/bin/spark-submit \
--class ${SPARK_APPLICATION_MAIN_CLASS} \
--master ${SPARK_MASTER_URL} \
--deploy-mode cluster \
--total-executor-cores 1 \
${SPARK_SUBMIT_ARGS} \
${SPARK_APPLICATION_JAR_LOCATION} \
${SPARK_APPLICATION_ARGS}
@mvillarrealb
mvillarrealb / spark-cluster-docker-compose.yml
Created September 23, 2018 17:37
A docker compose to create a spark standalone cluster using custom base images spark-master:2.3.1 and spark-worker:2.3.1
version: "3.7"
services:
spark-master:
image: spark-master:2.3.1
container_name: spark-master
hostname: spark-master
ports:
- "8080:8080"
- "7077:7077"
networks:
@mvillarrealb
mvillarrealb / build-images.sh
Created September 23, 2018 17:52
Script to build custom docker images spark-base, spark-master, spark-worker and spark-submit
#!/bin/bash
set -e
docker build -t spark-base:2.3.1 ./docker/base
docker build -t spark-master:2.3.1 ./docker/spark-master
docker build -t spark-worker:2.3.1 ./docker/spark-worker
docker build -t spark-submit:2.3.1 ./docker/spark-submit
@mvillarrealb
mvillarrealb / RecursiveScan.js
Created March 1, 2019 13:54
Scan recursively a directory to find all files, in the example we filter markdown(.md) files
const fs = require('fs');
const path = require('path');
/**
*
* @param {String} Directory to scan
* @param {Array} allFiles partial list of files(used for recursivity)
*/
const recursiveScan = function (directory, allFiles) {
const files = fs.readdirSync(directory);
allFiles = allFiles || [];
@mvillarrealb
mvillarrealb / recursiveScanAsync.js
Created March 1, 2019 14:23
Scans a directory in a recursive way using promises with async and await
const fs = require('fs');
const path = require('path');
const {
promisify
} = require('util');
/**
Add Promise sugar to the fs.readdir and fs.stat functions
*/
const fsReadDir = promisify(fs.readdir);
const fsStat = promisify(fs.stat);
@mvillarrealb
mvillarrealb / create_storage_class.sh
Created April 7, 2019 02:45
Create a storage class for a local kubernetes cluster
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
kubectl get storageclass
@mvillarrealb
mvillarrealb / install_helm.sh
Created April 7, 2019 02:46
Install helm on linux
#download helm
curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > install-helm.sh
#Make instalation script executable
chmod u+x install-helm.sh
#Install helm
./install-helm.sh