Skip to content

Instantly share code, notes, and snippets.

@coingraham
Created October 22, 2019 18:21
Show Gist options
  • Select an option

  • Save coingraham/38bb78c91d2ea98cf57f618494b1bb39 to your computer and use it in GitHub Desktop.

Select an option

Save coingraham/38bb78c91d2ea98cf57f618494b1bb39 to your computer and use it in GitHub Desktop.
Glue PySpark Command to run SQL
pre_query = "truncate table target_table"
post_query= "begin;delete from target_table using stage_table where stage_table.id = target_table.id ; insert into target_table select * from stage_table; drop table stage_table; end;"
connection_options_redshift = {
"preactions": pre_query,
"dbtable": "stage_table",
"database": "redshiftdb",
"postactions":post_query
}
datasink4 = glueContext.write_dynamic_frame.from_jdbc_conf(frame = datasource0, catalog_connection = "test_red",
connection_options = connection_options_redshift,
redshift_tmp_dir = 's3://s3path',
transformation_ctx = "datasink4")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment