Skip to content

Instantly share code, notes, and snippets.

@64lines
Last active February 19, 2019 20:03
Show Gist options
  • Save 64lines/ebee014be17a31e9ab4128db7ac308cd to your computer and use it in GitHub Desktop.
Save 64lines/ebee014be17a31e9ab4128db7ac308cd to your computer and use it in GitHub Desktop.
from pyspark.sql.functions import *
# Example 1
fact_df = fact_df.filter(col('colname').isNotNull()).alias('fact_df')
# Example 2
fact_df = fact_df.filter(col('colname').isNull()).alias('fact_df')
# Example 3
fact_df = fact_df.withColumn('colname', when(col('colname').isNotNull(), col('colname2')).otherwise(col('colname3'))).alias('fact_df')
# Example 4
fact_df = fact_df.withColumn('colname', when(col('colname').isNull(), col('colname2')).otherwise(col('colname3'))).alias('fact_df')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment