Roboflow async workflows often span multiple runtimes:
HTTP request / Node service -> Pub/Sub -> Python worker -> Pub/Sub -> Node completion handler
Cloud Trace handles a single request boundary, but Pub/Sub breaks that trace. Today, the durable link is usually a domain id embedded somewhere in each payload. In practice, that means an investigation has to know which field each service happened to log: jobId here, autoLabelJobId there, free text somewhere else. The result is brittle multi-clause log archaeology.
The fix is to make operation identity a first-class logging and propagation contract.
Every logical operation gets a stable operation_id. When a business id already exists, use it. For Autolabel, that is autoLabelJobId. Each runtime emits the same Cloud Logging labels:
operation_idoperation_typestageworkspace_id
These are labels, not arbitrary payload fields, so the query is payload-shape agnostic:
labels.operation_id="<id>"
Pub/Sub message attributes are the propagation carrier between services. Publishers stamp operation attributes from ambient context, and consumers seed their local logging context from those attributes. Propagation is gated on operation_id; request-only context such as workspace/stage without an operation id is not enough to create async correlation.
Node gets AsyncLocalStorage-backed operation context. HTTP and Pub/Sub entrypoints seed the context once; ordinary logger calls inherit it and emit the operation labels through the logging adapter.
Python already has thread-local logging context in roboflow-queues; the Pub/Sub processor adopts incoming operation attributes into that context instead of treating every message as an unrelated unit of work.
This keeps feature code mostly unchanged. Surface handlers only need a small context-enrichment call when the real operation id becomes known.
Autolabel is the first workflow using the contract end to end:
- trigger: seed
operation_id=autoLabelJobId,operation_type=autolabel,stage=trigger - worker: adopt the propagated operation attributes in
roboflow-queues - completion: enrich context with
stage=completiononce the parent Autolabel job is known
The expected investigation shape becomes:
labels.operation_id="<autoLabelJobId>"
That should return the operation timeline across Node trigger logs, Python worker logs, and Node completion logs.
Correlation answers “what happened to this operation?” Terminal events answer “how is this operation type doing overall?”
Terminal aggregate fields stay in jsonPayload:
status:success,partial_failure,failureduration_mssubjobs_totalsubjobs_failed
Those fields are intended for log-based metrics such as latency, success rate, failure rate, and partial-failure rate. Identity and grouping stay in labels; aggregate measurements stay in payload.
This is not a full OpenTelemetry migration and does not replace existing analytics events. It establishes the low-risk correlation layer first. Traceparent propagation and Cloud Trace waterfall stitching can layer on later.