Skip to content

Instantly share code, notes, and snippets.

@VinACE
Last active April 20, 2020 17:18
Show Gist options
  • Select an option

  • Save VinACE/2b4d0e6d13b7ba46f810bd2c3a8ec0bf to your computer and use it in GitHub Desktop.

Select an option

Save VinACE/2b4d0e6d13b7ba46f810bd2c3a8ec0bf to your computer and use it in GitHub Desktop.
docker-compose with hostnetwork connection
version: '3'
services:
spark:
image: bitnami/spark:2
network_mode: host
user: root
environment:
- SPARK_HOST=${SPARK_HOST}
- SPARK_MODE=master
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
ports:
- '8080:8080'
spark-worker-1:
image: bitnami/spark:2
network_mode: host
user: root
environment:
- SPARK_MODE=worker
- SPARK_MASTER_URL=spark://spark:7077
- SPARK_WORKER_MEMORY=1G
- SPARK_WORKER_CORES=1
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
spark-worker-2:
image: bitnami/spark:2
network_mode: host
user: root
environment:
- SPARK_MODE=worker
- SPARK_MASTER_URL=spark://spark:7077
- SPARK_WORKER_MEMORY=1G
- SPARK_WORKER_CORES=1
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
-e SPARK_DRIVER_HOST="ip address of the hosting host" -- this can be added into the environment variable...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment