- If values are integers in [0, 255], Parquet will automatically compress to use 1 byte unsigned integers, thus decreasing the size of saved DataFrame by a factor of 8.
- Partition DataFrames to have evenly-distributed, ~128MB partition sizes (empirical finding). Always err on the higher side w.r.t. number of partitions.
- Pay particular attention to the number of partitions when using
flatMap, especially if the following operation will result in high memory usage. TheflatMapop usually results in a DataFrame with a [much] larger number of rows, yet the number of partitions will remain the same. Thus, if a subsequent op causes a large expansion of memory usage (i.e. converting a DataFrame of indices to a DataFrame of large Vectors), the memory usage per partition may become too high. In this case, it is beneficial to repartition the output offlatMapto a number of partitions that will safely allow for appropriate partition memory sizes, based upon the
Pietro Berkes pberkes
dusenberrymw
/ spark_tips_and_tricks.md
Last active
June 28, 2024 12:37
Tips and tricks for Apache Spark.