Skip to content

Instantly share code, notes, and snippets.

@taion809

taion809/gist:9fa4b2a44b4740cd94700697ed2e4352

Created June 30, 2017 22:08
Show Gist options
  • Save taion809/9fa4b2a44b4740cd94700697ed2e4352 to your computer and use it in GitHub Desktop.
Save taion809/9fa4b2a44b4740cd94700697ed2e4352 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
gistfile1.txt
Output the state of the DAG when a task executes (output to logs)
Save the version of your workflow and the task code you used (so you can check logs later and recreate data as needed)
script a way to prune the data from any point in the DAG that will also prune data downstream so you can rerun workflows.
log task ids, hadoop application ids, etc when the task executes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment