swift tracing mini notes, sswg call

Swift (Baggage) Context & (Distributed) Tracing

Big Picture / Open Questions

"Context"

Everyone has context.

This is _Baggage_Context, intended specifically for carrying values over:

asynchronous boundaries

process boundaries
network boundaries

This is similar to Go's stdlib Context[1]:

	Deadline() (deadline time.Time, ok bool)
	Done() <-chan struct{}
	Err() error

	Value(key interface{}) interface{}
	
	// ...

BaggageContext is not FrameworkContext, as libraries already have context's:

NIO's ChannelHandlerContext,

Swift gRPC's UnaryResponseCallContext ,
Lambda Runtime's Lambda.Context`
Vapor's Request functions effectively as a context object,

public struct BaggageContext {
    public subscript<Key: BaggageContextKey>(_ key: Key.Type) -> Key.Value? // get/set
}

Unlike Go's style, we propose that nesting the baggage is fine:

// NIO
public final class BaggageContextInboundHTTPHandler: ChannelInboundHandler {
    public func channelRead(context: ChannelHandlerContext, data: NIOAny) {
        context.baggage
    }
}

// Lambda
Lambda.run { (request: Request, context, callback: @escaping (Result<Response, Error>) -> Void) in
  context.baggage.traceID // or context.traceID 
}

// Vapor
request.context // baggage context

[1] https://golang.org/pkg/context/

Baggage Context Passing & Impact on Library Design

We will try to favor simplicity and robustness rather than "magical propagation" (often seen in Java land).

This means, explcit propagation (most of the time), we worked on some rules about how to do so:

This means APIs need to take the shape of:

httpClient.get("http://...", context: context)

We strongly suggest to MAKE CONTEXT REQUIRED.

"dropped" context and traces is the single most problematic thing with tracing (!)

The possibility of wrappers

We may want to do additional libraries which "instrument" other async things.

let span = tracer.newSpan(operationName: "simple work", context: context)
defer { span.end() }

some()
work(context: span.context) 
here(context: span.context) // may have sub-spans

E.g. an "instrumented dispatch queue":

DispatchQueue.global.instrumented(operationName: "some-work", context: context).sync { 
  print("hello")
}

// start span: on submission
// end span: when closure completed

or futures:

nioFuture.instrumented(operationName: "some-maps", context: context) { 
  $0.map().map()
}
// start span: submission time
// end span: nested future completed

// various other APIs possible!

These are OUTSIDE OF THE SCOPE for the tracing API package. :-)

Argument naming/positioning

In order to propagate baggage through function calls (and asynchronous-boundaries it may often be necessary to pass it explicitly (unless wrapper APIs are provided which handle the propagation automatically).

When passing baggage context explicitly we strongly suggest sticking to the following style guideline:

Assuming the general parameter ordering of Swift function is as follows (except DSL exceptions):
1. Required non-function parameters (e.g. (url: String)),
2. Defaulted non-function parameters (e.g. (mode: Mode = .default)),
3. Required function parameters, including required trailing closures (e.g. (onNext elementHandler: (Value) -> ())),
4. Defaulted function parameters, including optional trailing closures (e.g. (onComplete completionHandler: (Reason) -> ()) = { _ in }).
Baggage Context should be passed as: the last parameter in the required non-function parameters group in a function declaration.

This way when reading the call side, users of these APIs can learn to "ignore" or "skim over" the context parameter and the method signature remains human-readable and “Swifty”.

Examples here: https://github.com/slashmo/gsoc-swift-baggage-context

Similar design to Swift-Metrics / Swift-Log

Global InstrumentationSystem, get an instrument or a tracer from it.

If you know what specific tracer you want, you can get that.

Open questions on inter-op with swift-log.

Not-only Tracing

We specifically choose to talk about "cross-cutting tools" on this layer, but perhaps it's too abstract without more examples of what we actually mean, so here's a few examples of what could be implemented as baggage instruments but is not "directly" distributed tracing:

deadlines

in a multi-service system, where a request has to go "through" n services from the edge, and the edge has a strict SLA of "must reply within 200ms", we may want to carry around the deadline value with the requests as they are propagated to downstream systems. ...
resource utilization / analysis / management in multi-tenant environments
- if may be useful to capture congestion of resources and "who is responsible for this overload".
authentication, delegation / access control / auditing
- This is not an area I'm an expert on but does come up as another use-case of such instruments; It feels right, since usually these also mean carrying along the execution of some task some identity information. I do not encourage building ad-hoc security if anyone ever gets to this, there's plenty literature about it, our only hope is that if such system needs to carry metadata, it should be able to use the same instrumentation "points" as tracers would. 😉
ad-hoc "propagate with this request"

But also Tracing

Tracing types in alignment with Open Telemetry spec.

The context is "our" baggage context.

// WORK IN PROGRESS

public protocol TracingInstrument: Instrument {
    var currentSpan: Span? { get }

    func startSpan(named operationName: String, baggage: BaggageContext, at timestamp: DispatchTime) -> Span
 }
 
   public protocol Span {
      var operationName: String { get }

      var startTimestamp: DispatchTime { get }
      var endTimestamp: DispatchTime? { get }

      var baggage: BaggageContext { get }

      mutating func addEvent(_ event: SpanEvent)

      mutating func end(at timestamp: DispatchTime)
 }

Tracers of interest:

X-Ray
Zipkin
Jaeger
Honeycomb
Instruments.app during development on a mac (could be fun?)
Any OpenTelemetry/Tracing compatible ones with Swift clients

Current Work In Progress

Logging and Context #37

Tracer API WIP gsoc-swift-tracing#62
NIO would depend on the context lib swift-nio #1574

ktoso/gist:4d160232407e4d5835b5ba700c73de37