Skip to content

Instantly share code, notes, and snippets.

@WillKoehrsen

WillKoehrsen/run_spark_parallel.py

Created September 24, 2018 20:39
Show Gist options
  • Save WillKoehrsen/c57d8003cb4c11c8d1c589b5011764a6 to your computer and use it in GitHub Desktop.
Save WillKoehrsen/c57d8003cb4c11c8d1c589b5011764a6 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
run_spark_parallel.py
# Run feature engineering in parallel
r = sc.parallelize(partitions, numSlices=N_PARTITIONS).\
map(lambda x: partition_to_feature_matrix(x, feature_defs)).collect()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment