Skip to content

Instantly share code, notes, and snippets.

@JD-V

JD-V/dataflow commands

Last active December 4, 2018 11:10
Show Gist options
  • Save JD-V/41c0dfeb4b4765ab8884bc53fbaa28f9 to your computer and use it in GitHub Desktop.
Save JD-V/41c0dfeb4b4765ab8884bc53fbaa28f9 to your computer and use it in GitHub Desktop.
Download ZIP
initiate data flow job
Raw
dataflow commands
create setup.py
user pip freeze to list all the depndencies and then add it to setup.py
activate virtual env
source ./bin/activate
to run dataflow
python df.py --project <gcp_proj_name> --input_topic projects/cldcvr-dev/topics/<topic-name> --runner DataflowRunner --temp_location gs://<bucket location for dumpig backup>/ --setup_file /Users/jd-v/datflow/setup.py --job_name <name_that_will_show_up_in_gcp>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment