lmolkova

Problem

Same operation may be instrumented multiple times (manual + auto or library-native + auto) because of multiple reasons (below). This usually affects protocol instrumentation (HTTP/gRCP/etc) as they are auto-instrumented already and quite popular. It manifests as duplicated spans that fights for context injection on RPC calls, double performance impact and increase telemetry bills. Here are some cases when it happens:

1. specialized instrumentation: native library instrumentaion can provide more rich data, better quality/performance + auto-instrumentation thats always on
2. new library behavior (user manually instrumented and then new version brings auto-instrumentation)
3. configuration error

While p1 is valid case, and p2/p3 are not, but we'd still would rather communicate the need to remove extra instrumentation instead of duplicating data.

Assumptions

lmolkova

       public class ResourceIdProcessor : ActivityProcessor
        {
            private  readonly AsyncLocal<string> resourceId = new AsyncLocal<string>();
            private readonly IHttpContextAccessor httpContextAccessor;

            public ResourceIdProcessor(IHttpContextAccessor httpContextAccessor)
            {
                resourceId = null;
            }

lmolkova

    public class ClientIpHeaderTelemetryInitializerCopy : TelemetryInitializerBase
    {
        private const string HeaderNameDefault = "X-Forwarded-For";
        private readonly char[] headerValuesSeparatorDefault = { ',' };

        private char[] headerValueSeparators;

        /// <summary>
        /// Initializes a new instance of the <see cref="ClientIpHeaderTelemetryInitializer" /> class.
        /// </summary>

lmolkova

Tracing API Comparison

A distributed trace is a set of events, triggered as a result of a single logical operation, consolidated across various components of an application. A distributed trace contains events that cross process, network and security boundaries. A distributed trace may be initiated when someone presses a button to start an action on a website - in this example, the trace will represent calls made between the downstream services that handled the chain of requests initiated by this button being pressed.

Contract difference

OpenTelemetry Tracing API is a very strict contract that enables tracing signal (not debugging or profiling). This contract is the same for all kinds of libraries and tracing backends and includes several related concepts:

• span creation, required and optional properties
• sampling
• exporting
• noop behavior in absence of tracing implementaiton
• extra information certain types spans should include (e.g. spans for http calls).

lmolkova

Short version

Context propagation protocol

application SDK MUST propagate trace parent part of the context (version, traceId, parentId and traceFlags) in the message in Diagnostic-Id application property following W3C trace-context format for traceparent encoding.

Also, propagate W3C trace context using W3C Trace Context Propagator from OpenTelemetry. It should populate traceparent and tracestate application properties. traceparent must match Diagnostic-Id value.

When extracting the context, first get W3C trace-context and then (if missing) read Diagnostic-Id.

lmolkova

using System;
using System.Diagnostics;
using Microsoft.Extensions.Logging;

namespace Microsoft.Extensions.Hosting
{
    public static class LoggerExtensions
    {
        public static IDisposable BeginWorkerScope(this ILogger logger)
        {

lmolkova

namespace AI

[Description("Link details.")]
struct Link
{
    [Description("A unique identifier for a span within a trace, assigned when the span is created.")]
    [MaxStringLength("256")]
    10: required  string	 id;

    [Description("A unique identifier for a trace. All spans from the same trace share the same traceId.")]

lmolkova

import "Domain.bond"

namespace AI

[Description("An instance of operation represents completion of an logical operation.")]
struct OperationData
    : Domain
{
    [Description("Schema version")]
    10: required int32 	 ver = 2;

lmolkova

Java Exception

Throwable property name	Proto property	Language-specific description
message	message	message that describes the current exception
cause	inner_errors	linked list of instances that caused the current exception
stackTrace	stack_frames	immediate frames on the call stack
suppressed	??? TODO	exceptions that were suppressed in order to deliver this exception
throwable.getClass()	error_info.error_type	type of Throwable

C# Exception

lmolkova

using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DependencyCollector;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Queue;

namespace filtering