Skip to content

Instantly share code, notes, and snippets.

View minyk's full-sized avatar
😀
Containerize! all the things!

Drake Youngkun Min minyk

😀
Containerize! all the things!
69 followers · 80 following
View GitHub Profile
@minyk
minyk / local-dir-hostpath.md
Last active May 21, 2020 07:28
Learning how to properly run Apache spark on the Kunernetes environments

Using spark.kubernetes.driver.volumes... configurations:

spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp1.mount.path=/tmp/data1
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp1.mount.readOnly=false
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp1.options.path=/data1
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp2.mount.path=/tmp/data2
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp2.mount.readOnly=false
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp2.options.path=/data2
@minyk
minyk / spark-hadoop-aws.md
Created May 22, 2020 13:30
Apache Spark and hadoop-aws(s3a) related resources
@minyk
minyk / spark-on-k8s.md
Last active May 11, 2022 01:26

Presentations and articles for Spark-on-K8S

For own self-education purpose

General

  1. Apache Spark on Kubernetes: A Technical Deep Dive - Yinan Li, Google, KubeCon + CloudNativeCon China 2018, Nov 15, 2018
  2. Ready to run! Get Started with Spark on Kubernetes, Jean-Yves Stephan, Spot.io, Feb 1, 2022
  3. Patterns and Anti-Patterns of running Apache bigdata projects in Kubernetes, Marton Elek, ApacheCon EU 2019, Oct 24, 2019