Created
September 7, 2017 15:21
-
-
Save gbraccialli/c7fa9318d95ed32e3387836730d15e80 to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| spark.master yarn | |
| spark.dynamicAllocation.enabled true | |
| spark.executor.memory 20G | |
| spark.executor.cores 4 | |
| spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version 2 | |
| spark.yarn.am.memory 2G | |
| spark.dynamicAllocation.cachedExecutorIdleTimeout 60s | |
| spark.yarn.executor.memoryOverhead 3G | |
| spark.dynamicAllocation.executorIdleTimeout 60s | |
| spark.driver.memory 10G | |
| spark.yarn.driver.memoryOverhead 2G | |
| spark.jars /mnt/lib/jars/graphframes-0.5.0-spark2.1-s_2.11.jar | |
| spark.driver.maxResultSize 4g | |
| spark.sql.shuffle.partitions 2000 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment