Skip to content

Instantly share code, notes, and snippets.

View existeundelta's full-sized avatar

ExisteUnDelta existeundelta

  • Barcelona
View GitHub Profile
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@existeundelta
existeundelta / pspark_config.py
Created February 28, 2018 08:47 — forked from robenalt/pspark_config.py
Sample pyspark context setting with configs params
# Set up spark configuration
conf = SparkConf().setMaster("yarn-client").setAppName("sparK-mer")
#conf = SparkConf().setMaster("local[16]").setAppName("sparK-mer")
conf.set("yarn.nodemanager.resource.cpu_vcores",args.C)
# Saturate with executors
conf.set("spark.executor.instances",executorInstances)
conf.set("spark.executor.heartbeatInterval","5s")
# cores per executor
conf.set("spark.executor.cores",args.E)
# set driver cores
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
FROM ubuntu:14.04
ENV SCALA_VERSION=2.10.4
ENV CASSANDRA_VERSION=2.2.3
ENV SPARK_CASSANDRA_CONNECTOR_VERSION=1.4.0
ENV CONFLUENT_VERSION=1.0.1
ENV ELASTICSEARCH_VERSION=1.7.3
ENV ELASTICSEARCH_SPARK_CONNECTOR_VERSION=2.1.2
ENV LOGSTASH_VERSION=2.0.0
ENV KIBANA_VERSION=4.2.0