Skip to content

Instantly share code, notes, and snippets.

@Mahdisadjadi

Mahdisadjadi/unpersist_rdd.md

Created May 20, 2022 02:24
Show Gist options
  • Save Mahdisadjadi/fb7b32b7bc801a5f6be9385b39a5ea23 to your computer and use it in GitHub Desktop.
Save Mahdisadjadi/fb7b32b7bc801a5f6be9385b39a5ea23 to your computer and use it in GitHub Desktop.
Download ZIP
unpersist RDDs with lost variable reference
Raw
unpersist_rdd.md

In Pyspark, you can use:

persisted_RDDs = spark.sparkContext._jsc.getPersistentRDDs()
for (i, rdd_) in persisted_RDDs.items():
    rdd_.unpersist()

spark is of type pyspark.sql.session.SparkSession.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment