Telemetry

What is OpenTelemetry?

OpenTracing   == traces            ( jaeger/tokio-tracing )
OpenCensus    == metrics           ( grafana/prometheus   )
OpenTelemetry == traces && metrics ( et al.               )

Tracing

Event Logging In Rust (wtf is `%` `?`)

The ? sigil is shorthand that specifies a field should be recorded using its fmt::Debug implementation.
The % sigil operates similarly, but indicates that the value should be recorded using its fmt::Display implementation.

From the docs: tracing - Rust

The ? sigil is shorthand that specifies a field should be recorded using its fmt::Debug implementation:
#[derive(Debug)]
struct MyStruct {
    field: &'static str,
}

let my_struct = MyStruct {
    field: "Hello world!"
};

// `my_struct` will be recorded using its `fmt::Debug` implementation.
event!(Level::TRACE, greeting = ?my_struct);
// is equivalent to:
event!(Level::TRACE, greeting = tracing::field::debug(&my_struct));
The % sigil operates similarly, but indicates that the value should be recorded using its fmt::Display implementation:
   // `my_struct.field` will be recorded using its `fmt::Display` implementation.
   event!(Level::TRACE, greeting = %my_struct.field);
   // is equivalent to:
   event!(Level::TRACE, greeting = tracing::field::display(&my_struct.field));
The % and ? sigils may also be used with local variable shorthand:
   // `my_struct.field` will be recorded using its `fmt::Display` implementation.
   event!(Level::TRACE, %my_struct.field);
Additionally, a span may declare fields with the special value Empty, which indicates that that the value for that field does not currently exist but may be recorded later. For example:
   use tracing::{trace_span, field};
   
   // Create a span with two fields: `greeting`, with the value "hello world", and
   // `parting`, without a value.
   let span = trace_span!("my_span", greeting = "hello world", parting = field::Empty);
   
   // ...
   
   // Now, record a value for parting as well.
   span.record("parting", &"goodbye world!");
Finally, events may also include human-readable messages, in the form of a format string and (optional) arguments, after the event’s key-value fields. If a format string and arguments are provided, they will implicitly create a new field named message whose value is the provided set of format arguments.

preferred method of reporting events is to use the macros and key-value pairs. the most common macros include debug!() info!(). debug is more verbose and info is meant for emitting important happy path events.

use key value pairs. so when reporting an info you want to do it like this: info!(key1 = value1, key2 = value2, message = "this transaction is done!")

Directives:

The tokio/tracing crate uses the environmental variable RUST_LOG to pass in a “Directive” string for log filtering: target[span{field=value}]=level for all workspace members:

To show all errors: RUST_LOG=error
To show errors from workspace-member1 target: RUST_LOG="workspace_member1=error"
To show errors from workspace-member2 on the data span (correlating to this method: RUST_LOG="workspace_member2[data]=error"
To show errors for all above and filter where the sender is bob: RUST_LOG='workspace_member2[data{sender="bob"}]=error'

More info here

Spans

warn_span!("warn").in_scope(|| {
    info_span!("info").in_scope(|| {
        warn!("I AM WARN");
        info!("I AM INFO");
    });
});

Setting trace level to INFO preserves both the the info span and event:

$ RUST_LOG=info cargo run

 WARN warn:info: I AM WARN
 INFO warn:info: I AM INFO

Debugging

To only run something during cargo run rather than form the binary:

#[cfg(debug_assertions)]
{
    use opentelemetry::trace::TraceContextExt;
    if !parent_ctx.span().span_context().is_valid() {
        warn!("invalid span context, unable to get parent context");
    }
}

But setting the trace level to WARN has the "info" span omitted and the event inherited by the parent span:

$ RUST_LOG=warn cargo run

WARN warn: I AM WARN

mkatychev/telemetry.md