PySpark DataFrame filtering using a UDF and Regex

pyspark_udf_filtering.py

from pyspark.sql.functions import udf
from pyspark.sql.types import BooleanType

def regex_filter(x):
    regexs = ['.*ALLYOURBASEBELONGTOUS.*']
    
    if x and x.strip():
        for r in regexs:
            if re.match(r, x, re.IGNORECASE):
                return True
    
    return False 
    
    
filter_udf = udf(regex_filter, BooleanType())

df_filtered = df.filter(filter_udf(df.field_to_filter_on))

This was very useful, cheers!

thank you for sharing

This is very useful, Thanks

This was very useful, Thanks

Cool, this is exactly what I was looking for. Thanks.

thank you, this was helpful as its difficult to get sample codes on pyspark.

Thanks , simple and straight foward

👍 @huzefa