The dataflow
package and tooling (./scripts/dataflow
) serves a few purposes:
- makes it easier (and possible!) to write, test, and run Dataflow (Apache Beam) jobs locally and on GCP;
- formalizes some patterns and provides some structure to jobs;
- ensures any data modifications are logged.
Our typical use cases for Dataflow jobs include: