minyk

Presentations and articles for Spark-on-K8S

For own self-education purpose

General

1. Apache Spark on Kubernetes: A Technical Deep Dive - Yinan Li, Google, KubeCon + CloudNativeCon China 2018, Nov 15, 2018
2. Ready to run! Get Started with Spark on Kubernetes, Jean-Yves Stephan, Spot.io, Feb 1, 2022
3. Patterns and Anti-Patterns of running Apache bigdata projects in Kubernetes, Marton Elek, ApacheCon EU 2019, Oct 24, 2019

minyk

Apache Spark and AWS S3 https://medium.com/nmc-techblog/spark-dynamic-partition-inserts-part-1-5b66a145974f https://medium.com/nmc-techblog/spark-dynamic-partition-inserts-and-aws-s3-part-2-9ba0c97ad2c0

https://www.slideshare.net/databricks/apache-spark-at-scale-in-the-cloud

https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html

minyk

Using spark.kubernetes.driver.volumes... configurations:

spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp1.mount.path=/tmp/data1
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp1.mount.readOnly=false
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp1.options.path=/data1
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp2.mount.path=/tmp/data2
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp2.mount.readOnly=false
spark.kubernetes.executor.volumes.hostPath.spark-local-dir-tmp2.options.path=/data2

minyk

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  rules:
  - host: dashboard.192.168.100.12.nip.io
    http:
      paths:

minyk

---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: guestbook-back
spec:
  replicas: 1
  selector:
    matchLabels:
      type: app

minyk

Custom S3 endpoints with Spark

To be able to use custom endpoints with the latest Spark distribution, one needs to add an external package (hadoop-aws). Then, custum endpoints can be configured according to docs.

Use the hadoop-aws package

bin/spark-shell --packages org.apache.hadoop:hadoop-aws:2.7.2

SparkContext configuration

minyk

Papers (and articles) for the Container Technologies

For own self-education purpose

Containers

1. TBD
2. Docker: Lightweight Linux Containers for Consistent Development and Deployment, Dirk Merkel, Linux Journal, March 2014
3. Networking in Containers and Container Clusters, Victor Marmol, Rohit Jnagal, and Tim Hockin, netdev 0.1, Feb 14-17, 2015

minyk

{
  "id": "/mw/memory-stat",
  "cpus": 1,
  "mem": 2048,
  "instances": 1,
  "container": {
    "type": "DOCKER",
    "docker": {
      "forcePullImage": true,
      "image": "harbor.ajway.kr/middleware/mw:1.0",

minyk

build

• Get source

$ go get -u github.com/cendhu/fetch-block

• Get govendor

$ go get -u github.com/kardianos/govendor

• build

minyk

tadpole_1  | 2017-12-01 08:38:59,033 ERROR - TadpoleSystem_UserQuery : saveLoginHistory save login history 
tadpole_1  | com.ibatis.common.jdbc.exception.NestedSQLException:   
tadpole_1  | --- The error occurred in com/hangum/tadpole/engine/query/internal/system/Tadpole-System-Common.xml.  
tadpole_1  | --- The error occurred while executing update.  
tadpole_1  | --- Check the   INSERT INTO login_history(user_seq, login_ip, succes_yn, fail_reason)   VALUES (?, ?, ?, ?); .  
tadpole_1  | --- Check the SQL Statement (preparation failed).  
tadpole_1  | --- Cause: java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (no such table: login_history)
tadpole_1  | 	at com.ibatis.sqlmap.engine.mapping.statement.MappedStatement.executeUpdate(MappedStatement.java:107)
tadpole_1  | 	at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.insert(SqlMapExecutorDelegate.java:393)
tadpole_1  | 	at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.insert(SqlMapSessionImpl.java:82)