Malthe Borch malthe
chhantyal
/ spark_rdd_to_pandas_distributed.py
Last active
April 27, 2023 23:53
Convert Spark RDD to Pandas DataFrame inside Spark executors and make Spark DataFrame from resulting RDD. This is distributed i.e. no need for collecting RDD to driver.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """ | |
| Spark DataFrame is distributed but it lacks many features compared to Pandas. | |
| If you want to use Pandas, you can't just convert Spark DF to Pandas because that means collecting it to driver. | |
| It can be slow & not work at all when data size is big. | |
| So only way to use Pandas is to create mini dataframes inside executors. | |
| This gist shows how to create DataFrame from RDD inside Spark executors & build Spark DataFrame from final output. | |
| """ | |
| # Convert function to use in mapPartitions |