Skip to content

Instantly share code, notes, and snippets.

@atamborrino

atamborrino/sparkEc.scala

Last active December 10, 2020 16:01
Show Gist options
  • Save atamborrino/e5f6ce701e0200505f9f647473d4ce9c to your computer and use it in GitHub Desktop.
Save atamborrino/e5f6ce701e0200505f9f647473d4ce9c to your computer and use it in GitHub Desktop.
Download ZIP
Serializable Scala ExecutionContext for Spark. Allows to automatically re-create a thread-pool per Spark worker.
Raw
sparkEc.scala
class ECProvider()(implicit conf: Config) extends Serializable {
@transient implicit lazy val ec: ExecutionContext = {
ExecutionContext.fromExecutorService(
Executors.newWorkStealingPool(conf.getForkJoinPoolMaxParallelism())
)
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment