Dataflow data generator

dataflow.md

Google Cloud Blog post:

• Synthetic data generation with Dataflow data generator flex template

More detail on the Streaming Data Generator template:

• Streaming Data Generator to Pub/Sub, BigQuery, and Cloud Storage

The source code for the template:

• Dataflow Streaming Data Generator

It appears to be using json-data-generator library. Its GitHub repo has docs on how to define the schema:

• vincentrussell/json-data-generator

In streaming mode, it generates an unbounded input collection using Beam's GenerateSequence PTransform.

Interestingly enough, the documentation for that PTransform states

A PTransform that produces longs starting from the given value, and either up to the given limit or until Long.MAX_VALUE / until the given time elapses.

My interpretation of this is that eventually it will run out of values to produce?

Oracle states that

The long data type is a 64-bit two's complement integer. The signed long has a minimum value of -2^63 and a maximum value of 2^63-1.

Which is a large number, but still finite. This is not an issue, more of a curiousity, so I'll park this study for now.