Skip to content

Instantly share code, notes, and snippets.

@taiwotman

taiwotman/aws_redshift_pyspark2.py

Created December 18, 2020 00:51
Show Gist options
  • Save taiwotman/55d8f6904f65d1423eee626ee2f3959f to your computer and use it in GitHub Desktop.
Save taiwotman/55d8f6904f65d1423eee626ee2f3959f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
aws_redshift_pyspark2.py
#Transformation
df = df.withColumn("latitude", col("latitude").cast(DoubleType()))\
.withColumn("longitude", col("longitude").cast(DoubleType()))
df.printSchema()
df.show(10)
#More Insight
df.filter(df.incident_number== 'F190058688').show()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment