Fixlogs cross-service operation correlation rollout

fixlogs-cross-service-correlation.md

Fixlogs: Cross-Service Operation Correlation

This stack adds a shared correlation primitive for async operations that cross Node services, Pub/Sub, and roboflow-queues.

Goal

Make one logical operation queryable with one field across every runtime:

labels.operation_id="<operation id>"

Autolabel is the pilot, but the contract is intended for other async workflows too.

Contract

Operation identity is emitted as Cloud Logging labels:

• operation_id: stable id for the logical operation, usually the business/job id
• operation_type: operation family, e.g. autolabel
• stage: current hop, e.g. trigger, worker, completion
• workspace_id: tenant context when available

Pub/Sub propagation uses message attributes. Propagation is intentionally gated on operation_id; ordinary request/workspace context without an operation id is not stamped onto async work.

Terminal aggregate fields stay in jsonPayload, not labels:

• status: success, partial_failure, failure
• duration_ms
• subjobs_total
• subjobs_failed

PR Stack

1. roboflow#12728 — Node operation context foundation

   • Adds AsyncLocalStorage-backed operation context.
   • Emits active context as Cloud Logging labels through the Node logging adapter.
   • Seeds request-scoped context in HTTP logging middleware.

2. roboflow#12729 — Node Pub/Sub propagation

   • Stamps operation labels into Pub/Sub attributes at the shared publish chokepoints.
   • Seeds Node Pub/Sub consumers from attributes.
   • Preserves existing queue call shapes and caller-provided attributes.

3. roboflow-queues#113 — Python worker propagation

   • Adopts incoming Pub/Sub operation attributes in roboflow-queues.
   • Feeds the existing Python thread-local logging context.
   • Propagates operation attributes on completion messages.

4. roboflow#12731 — Autolabel pilot wiring

   • Seeds autolabel trigger context after autoLabelJobId is minted.
   • Seeds completion context after the Autodistill subjob identifies the parent Autolabel job.
   • Uses small setOperationContext(...) calls rather than wrapping large handlers.

5. roboflow#12730 — Rollout docs / definition of done

   • Documents the cross-service logging contract in the observability recipe.
   • Captures query examples and terminal-event guidance for aggregate metrics.

Expected Result

The old Autolabel investigation shape was a multi-clause query over different payload fields. The new shape is:

labels.operation_id="<autoLabelJobId>"

That should return the trigger, worker, and completion logs regardless of service/runtime payload shape.

This is intentionally incremental: correlation labels and Pub/Sub attributes first; Cloud Trace/traceparent stitching can layer on later.