Skip to content

Instantly share code, notes, and snippets.

@lmolkova

lmolkova/distro_arch_board.md

Last active August 1, 2023 20:21
Show Gist options
  • Save lmolkova/8a1878e17532658d779789e071b1262e to your computer and use it in GitHub Desktop.
Save lmolkova/8a1878e17532658d779789e071b1262e to your computer and use it in GitHub Desktop.
Download ZIP
Azure Monitor Distro and Exporter GA review
Raw
distro_arch_board.md

Scope

GA review for

  • Azure Monitor OpenTelemetry disto in .NET, Python, and JS.
    • desired GA date Aug 18
  • Azure Monitor OpenTelemetry Exporters in .NET, Python, JS and Java
    • desired GA date: .NET - Aug 18, Java - by the end of 2023, Python, JS - TBD

Hero scenarios

There are two different ways to collect and send OpenTelemetry data to Azure Monitor:

  • distribution (aka distro) - a customized version of OTel with predefined set of instrumentation and configurations. Minimal configuration is required from the user side.
  • exporter - user picks instrumentations, configures OpenTelemetry directly, and enables specific exporter

We'll take a look at both and see how to apply basic configuration to them.

See also Application Insights learn,microsoft.com articles for OpenTelemetry.

Python

Distro

pip install azure-monitor-opentelemetry --pre

Configure auto-collection for traces, metrics, and logs.

from azure.monitor.opentelemetry import configure_azure_monitor

configure_azure_monitor()

This configuration enables collection of the following data:

  • HTTP server-side: Django, Flask, FastAPI (traces, metrics)
  • HTTP client side: requests, urllib, urllib3 (traces, metrics)
  • DB: psycopg2 (traces, metrics)
  • Azure SDKs (traces)

By default, connection string is obtained from APPLICATIONINSIGHTS_CONNECTION_STRING environment variable.

Configuration

Users can configure the distro using environment variables and/or configuration options.

Enable AAD
from azure.monitor.opentelemetry import configure_azure_monitor
configure_azure_monitor(
    credential=ManagedIdentityCredential(),
)
Configure sampling
export OTEL_TRACES_SAMPLER_ARG=0.1
from azure.monitor.opentelemetry import configure_azure_monitor
configure_azure_monitor()

Can also be done with OTEL_TRACES_SAMPLER_ARG environment variable.

Tune OpenTelemetry configuration
pip install opentelemetry-exporter-otlp
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

from azure.monitor.opentelemetry import configure_azure_monitor

configure_azure_monitor()
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317")
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)
Enable 3rd party instrumentations or report custom traces and metrics
pip install opentelemetry-instrumentation-sqlalchemy --pre
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from logging import getLogger

configure_azure_monitor()
SQLAlchemyInstrumentor().instrument(...)

tracer = trace.get_tracer(__name__)
logger = getLogger(__name__)
with tracer.start_as_current_span("test"):
    logger.info("Correlated info log")

Logging is supported through logging handler.

See also distro README

Following configuration options are supported through environment variables (defined by OpenTelemetry) and/or kwargs:

  • Connection String
  • Cloud Role Name
  • Cloud Role Instance
  • Sample rate
  • AAD Auth
  • Offline Storage
  • Tracing export interval
  • Logging export interval
  • Disabling instrumentations

Exporter

Users who want to have full control over telemetry collection configuration, can configure OpenTelemetry and just export their telemetry to Azure Monitor.

pip install azure-monitor-opentelemetry-exporter --pre
pip install opentelemetry-instrumentation-requests --pre
from opentelemetry import trace
from opentelemetry.instrumentation.requests import RequestInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from azure.monitor.opentelemetry.exporter import (
    ApplicationInsightsSampler,
    AzureMonitorTraceExporter,
)

sampler = ApplicationInsightsSampler(0.75)
trace.set_tracer_provider(TracerProvider(sampler=sampler))

exporter = AzureMonitorTraceExporter(
    connection_string=connection-string
)
span_processor = BatchSpanProcessor(exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

RequestInstrumentor.instrument()
Traces, metrics, and logs exporters
from azure.monitor.opentelemetry.exporter import (
	AzureMonitorTraceExporter,
	AzureMonitorMetricExporter,
	AzureMonitorLogExporter,
)

trace_exporter = AzureMonitorTraceExporter()
metric_exporter = AzureMonitorMetricExporter()
log_exporter = AzureMonitorLogExporter()

See also exporter README

.NET

Distro

dotnet add package Azure.Monitor.OpenTelemetry.AspNetCore --prerelease

Configure auto-collection for traces, metrics, and logs:

using Azure.Monitor.OpenTelemetry.AspNetCore;

var builder = WebApplication.CreateBuilder(args);
builder.Services.AddOpenTelemetry().UseAzureMonitor();
//...

This configuration enables collection of the following data:

  • HTTP client and ASP.NET Core (all signals)
  • DB: SQL client (traces)
  • Azure SDK (traces, still requires user to opt in with feature flag)

By default, connection string is obtained from APPLICATIONINSIGHTS_CONNECTION_STRING environment variable - staying consistent with Track 1 SDK.

Configuration

Users can configure the distro explicitly in-code, via Microsoft.Extensions.Configuration, and with a few environment variables.

Enable AAD
builder.Services.AddOpenTelemetry().UseAzureMonitor(o =>
{
    o.ConnectionString = "InstrumentationKey=00000000-0000-0000-0000-000000000000";
    o.Credential = new DefaultAzureCredential();
});
Configure sampling
builder.Services.AddOpenTelemetry().UseAzureMonitor(o =>
{
    o.SamplingRatio = 0.5F;
});
Tune OpenTelemetry instrumentation libraries
builder.Services.AddOpenTelemetry().UseAzureMonitor();
builder.Services.Configure<AspNetCoreInstrumentationOptions>(options =>
{
    options.RecordException = true;
    options.Filter = (httpContext) =>
    {
        // only collect telemetry about HTTP GET requests
        return HttpMethods.IsGet(httpContext.Request.Method);
    };
});

Note that AspNetCoreInstrumentationOptions is defined in OpenTelemetry.Instrumentation.AspNetCore package.

Enable 3rd party instrumentations or report custom traces and metrics
dotnet add package OpenTelemetry.Instrumentation.GrpcNetClient --prerelease
builder.Services.AddOpenTelemetry().UseAzureMonitor();
builder.Services.ConfigureOpenTelemetryTracerProvider((sp, builder) =>
    builder.AddGrpcClientInstrumentation()
           .AddSource("MyCompany.MyProduct.MyLibrary"));
builder.Services.ConfigureOpenTelemetryMeterProvider((sp, builder) => builder.AddMeter("MyCompany.MyProduct.MyLibrary"));

Logging is supported through Microsoft.Extensions.Logging, so all the configuration is done with corresponding options and outside of the distro.

See also distro README

Exporter

Users who don't use ASP.NET Core or who want to have full control over telemetry collection configuration, can configure OpenTelemetry and just export their telemetry to Azure Monitor.

dotnet add package Azure.Monitor.OpenTelemetry.Exporter --prerelease
dotnet add package OpenTelemetry.Instrumentation.Http  --prerelease
dotnet add package OpenTelemetry.Exporter.Console
var resourceAttributes = new Dictionary<string, object>
{
    { "service.name", "my-service" },
    { "service.namespace", "my-namespace" },
    { "service.instance.id", "my-instance" },
};

var resourceBuilder = ResourceBuilder.CreateDefault().AddAttributes(resourceAttributes);

_tracerProvider = Sdk.CreateTracerProviderBuilder()
                .SetResourceBuilder(resourceBuilder)
                .AddSource("my source")
                .AddHttpClientInstrumentation()
                .AddProcessor(new ActivityFilteringProcessor())
                .AddConsoleExporter()
                .AddAzureMonitorTraceExporter(o => { o.ConnectionString = connectionString; o.SamplingRatio = 1.0F; }, credential)
                .Build();

Only AddAzureMonitorTraceExporter and it's parameters are part of Azure Monitor Exporter, the rest of this code is vanilla OpenTelemetry configuration.

Traces, metrics, and logs exporters
Sdk.CreateTracerProviderBuilder()
    .AddAzureMonitorTraceExporter()
    .Build();

Sdk.CreateMeterProviderBuilder()
    .AddAzureMonitorMetricExporter()
    .Build();

LoggerFactory.Create(builder =>
{
    builder.AddOpenTelemetry(options =>
    {
        options.AddAzureMonitorLogExporter();
    });
});

See also exporter README

JS

Distro

npm install @azure/monitor-opentelemetry
const { AzureMonitorOpenTelemetryClient, AzureMonitorOpenTelemetryOptions } = require("@azure/monitor-opentelemetry");

const options: AzureMonitorOpenTelemetryOptions = {
  azureMonitorExporterConfig: {},
}
const azureMonitorClient = new AzureMonitorOpenTelemetryClient(options);

This configuration enables collection of the following data:

  • HTTP: Node.js http/https (traces, metrics)
  • DB: mongodb, mysql, redis, postgres (traces)
  • Azure SDKs (traces)
  • standard cpu/mem usage counters

By default, connection string is obtained from APPLICATIONINSIGHTS_CONNECTION_STRING environment variable.

Configuration

Users can configure distro through explicit configuration options or configuration file.

Enable AAD
const { ApplicationInsightsClient, ApplicationInsightsConfig } = require("applicationinsights");
const { ManagedIdentityCredential } = require("@azure/identity");

const credential = new ManagedIdentityCredential();

const config = new ApplicationInsightsConfig();
config.azureMonitorExporterConfig.aadTokenCredential = credential;
const appInsights = new ApplicationInsightsClient(config);
Configure sampling
const { ApplicationInsightsClient, ApplicationInsightsConfig } = require("applicationinsights");
const config = new ApplicationInsightsConfig();
config.samplingRatio = 0.75;
const appInsights = new ApplicationInsightsClient(config);
Tune OpenTelemetry configuration
const { AzureMonitorOpenTelemetryClient } = require("@azure/monitor-opentelemetry");
const { ReadableSpan, SpanProcessor } = require("@opentelemetry/sdk-trace-base");

const azureMonitorClient = new AzureMonitorOpenTelemetryClient();

class SpanEnrichingProcessor implements SpanProcessor{
    //...
    onEnd(span: ReadableSpan){
        span.attributes["CustomDimension1"] = "value1";
        span.attributes["CustomDimension2"] = "value2";
    }
}

azureMonitorClient.getTracerProvider().addSpanProcessor(new SpanEnrichingProcessor());
Enable 3rd party instrumentations or report custom traces and metrics
npm install @opentelemetry/instrumentation-express
const { AzureMonitorOpenTelemetryClient } = require("@azure/monitor-opentelemetry");
const { registerInstrumentations } = require("@opentelemetry/instrumentation");
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');

const azureMonitorClient = new AzureMonitorOpenTelemetryClient();
const instrumentations = [
    new ExpressInstrumentation(),
];
registerInstrumentations({
    tracerProvider: azureMonitorClient.getTracerProvider(),
    meterProvider: azureMonitorClient.getMeterProvider(),
    instrumentations: instrumentations,
});

See also distro README

Exporter

Users who want to have full control over telemetry collection configuration, can configure OpenTelemetry and just export their telemetry to Azure Monitor.

npm install @azure/monitor-opentelemetry
npm install @opentelemetry/instrumentation-http
const { BatchSpanProcessor } = require("@opentelemetry/sdk-trace-base");
const { NodeTracerProvider } = require("@opentelemetry/sdk-trace-node");
const { registerInstrumentations } = require('@opentelemetry/instrumentation');
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
const { ApplicationInsightsSampler, AzureMonitorTraceExporter} = require("@azure/monitor-opentelemetry-exporter");
const { Resource } = require("@opentelemetry/resources");

const aiSampler = new ApplicationInsightsSampler(0.75);
const provider = new NodeTracerProvider({
  sampler: aiSampler,
  resource: new Resource({
    ["service.name"]: "basic-service",
  }),
});

registerInstrumentations({
  instrumentations: [
    new HttpInstrumentation()
  ],
});

const trace_exporter = new AzureMonitorTraceExporter();
trace_provider.addSpanProcessor(
  new BatchSpanProcessor(exporter, {
    bufferTimeout: 15000,
    bufferSize: 1000
  })
);

provider.register();
Traces and metrics exporters
const { AzureMonitorMetricExporter, AzureMonitorTraceExporter} = require("@azure/monitor-opentelemetry-exporter");

const trace_exporter = new AzureMonitorTraceExporter();
const metric_exporter = new AzureMonitorMetricExporter();

See also exporter README

Java

Distro

Not in the scope of this review.

java -javaagent:applicationinsights-agent-3.4.15.jar -jar ....

Supports configuration with OTEL_* and APPLICATIONINSIGHT_* environment variables and configuration file. AAD is supported with `AZURE_*`` identity env vars.

Auto-collects a variety of signals from Java libraries: HTTP, GRPC, databases, messaging systems, Azure SDKs, common logging libraries, common metrics, etc. Supports custom telemetry collection - see distro docs.

Exporter

<dependency>
  <groupId>com.azure</groupId>
  <artifactId>azure-monitor-opentelemetry-exporter</artifactId>
</dependency>
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.metrics.export.MetricExporter;
import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader;
import io.opentelemetry.sdk.metrics.SdkMeterProvider;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.data.SpanData;
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
import io.opentelemetry.sdk.trace.export.SpanExporter;

import com.azure.monitor.opentelemetry.exporter.*;

SpanExporter traceExporter = new AzureMonitorExporterBuilder()
    .buildTraceExporter();

MetricExporter metricExporter = new AzureMonitorExporterBuilder()
    .buildMetricExporter();

// otel config
SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
    .addSpanProcessor(SimpleSpanProcessor.create(exporter))
    .build();
PeriodicMetricReader periodicMetricReader = PeriodicMetricReader
    .builder(exporter)
    .build();
SdkMeterProvider sdkMeterProvider = SdkMeterProvider.builder()
    .registerMetricReader(periodicMetricReader)
    .build();

OpenTelemetry openTelemetry = OpenTelemetrySdk.builder()
    .setTracerProvider(tracerProvider)
    .setMeterProvider(sdkMeterProvider)
    .buildAndRegisterGlobal();

See also exporter README

API views

@TimothyMothra
Copy link

typo: "environemnt variable"

@TimothyMothra
Copy link

typo: "See also Application Insights learn,microsoft.com articles for OpenTelemetry."

The link is correct, but the text has a comma instead of a period.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment