Skip to content

Instantly share code, notes, and snippets.

@nmukerje
Created February 8, 2021 22:49
Show Gist options
  • Save nmukerje/98907a6402d5fb14c95809e5fa1d92b7 to your computer and use it in GitHub Desktop.
Save nmukerje/98907a6402d5fb14c95809e5fa1d92b7 to your computer and use it in GitHub Desktop.
Glue Studio SQL Transform
def SQLTransform (glueContext, dfc) -> DynamicFrameCollection:
dfc.select(list(dfc.keys())[0]).toDF().createOrReplaceTempView("<SRC1>")
### Enter your SQL Statement here
sql_statement = "SELECT bucket,key,struct.col1,array_col[0].array_col1[0].col1 FROM <SRC1> a ..."
###
output_df = spark.sql(sql_statement)
dyf_output = DynamicFrame.fromDF(output_df, glueContext, "SQLTransform0")
return(DynamicFrameCollection({"SQLTransform0": dyf_output}, glueContext))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment