Skip to content

Instantly share code, notes, and snippets.

@barograf

barograf/job_4.py

Last active November 4, 2022 09:40
Show Gist options
  • Save barograf/aebc4288f49d29f7ec8d017f1fe36a76 to your computer and use it in GitHub Desktop.
Save barograf/aebc4288f49d29f7ec8d017f1fe36a76 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
job_4.py
source_df = source_dynf.toDF()
partitioned_df = source_df.withColumn(
"year", functions.year(functions.col("created_at"))
).withColumn("month", functions.month(functions.col("created_at")))
partitioned_dynf = DynamicFrame.fromDF(
partitioned_df, glueContext, f"partitioned_dynf_{tbl_name}"
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment