Skip to content

Instantly share code, notes, and snippets.

@d4rkc0de

d4rkc0de/Spark conf to avoid OOM on Overhead

Last active November 7, 2022 17:39
Show Gist options
  • Save d4rkc0de/cc407e741f9a2b5ae688dd220e9f8a63 to your computer and use it in GitHub Desktop.
Save d4rkc0de/cc407e741f9a2b5ae688dd220e9f8a63 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Spark conf to avoid OOM on Overhead
// case FetchFailedException or MetadataFetchFailedException: how to avoid BroadcastNestedLoopJoin
- spark.executor.memoryOverhead=1g
- spark.kubernetes.memoryOverheadFactor=0.2
// avoid skew spark >= 3.0
spark.sql.adaptive.optimizeSkewedJoin.enabled
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment