Skip to content

Instantly share code, notes, and snippets.

@barograf

barograf/job_5.py

Last active November 4, 2022 09:40
Show Gist options
  • Save barograf/bc0dc16f5c38d802bb1ad865fb924f5f to your computer and use it in GitHub Desktop.
Save barograf/bc0dc16f5c38d802bb1ad865fb924f5f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
job_5.py
sink = glueContext.getSink(
connection_type="s3",
path=f"s3a://{dest_path}",
enableUpdateCatalog=True,
updateBehavior="UPDATE_IN_DATABASE",
partitionKeys=["year", "month"],
)
sink.setFormat("glueparquet")
sink.setCatalogInfo(
catalogDatabase=args["CATALOG"],
catalogTableName=f"{args['DEST_TABLE_PREFIX']}{tbl_name}",
)
sink.writeFrame(partitioned_dynf)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment