Skip to content

Instantly share code, notes, and snippets.

@mheilman

mheilman/pyspark_pandas_udf_sklearn.ipynb

Last active July 8, 2022 12:53
Show Gist options
  • Save mheilman/6ce261549b55bf4997ec102ad4e8d643 to your computer and use it in GitHub Desktop.
Save mheilman/6ce261549b55bf4997ec102ad4e8d643 to your computer and use it in GitHub Desktop.
Download ZIP
example for "Prediction at Scale with scikit-learn and PySpark Pandas UDFs" (https://medium.com/civis-analytics/prediction-at-scale-with-scikit-learn-and-pyspark-pandas-udfs-51d5ebfb2cd8)
Display the source blob
Display the rendered blob
Raw
pyspark_pandas_udf_sklearn.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@jamesonl
Copy link

jamesonl commented Dec 3, 2019

Hi - this code is helpful for applying an already trained model at scale... but is it possible to train a model at scale using pandas_UDF functions?

Another way of asking the same question: Is it possible to include the section called "Train a model with scikit-learn" within a pandas_UDF?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment