Last active
July 15, 2023 13:23
-
-
Save zoltanctoth/2deccd69e3d1cde1dd78 to your computer and use it in GitHub Desktop.
Writing an UDF for withColumn in PySpark
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from pyspark.sql.types import StringType | |
from pyspark.sql.functions import udf | |
maturity_udf = udf(lambda age: "adult" if age >=18 else "child", StringType()) | |
df = spark.createDataFrame([{'name': 'Alice', 'age': 1}]) | |
df.withColumn("maturity", maturity_udf(df.age)) | |
df.show() |
It was nice to come across my teacher's code even after graduation. Thank you!
I encountered this problem too. Have you solved it? Thank you!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
TypeError: a bytes-like object is required, not 'NoneType'
I am getting this error while trying 'mrandrewandrade' input.
How can I resolve this error?
Thanks.