Introduction

I just found the following article in the OpenTelemetry blog

Announcing a Community Demo for OpenTelemetry
https://opentelemetry.io/blog/2022/demo-announcement/

This is a demo kit for OpenTelemetry, where an e-commerce site is built with microservices in various languages. In addition, metrics and traces are already instrumented, and if you have a Docker environment at hand, you can just docker compose up to get it running. As a demonstration of OpenTelemetry, you can see traces and metrics in Jeager and Prometheus. In this article, I would like to send them to Datadog so that I can see them in Datadog.

Configuration

It consists of several microservices, each in a different language. (Image taken from blog post.)

How to run the demo

Clone the Github repository and docker compose up to get it running. It will take a while to build the first time.

Webstore Demo
https://github.com/open-telemetry/opentelemetry-demo-webstore

Once up and running, access http://localhost:8080 in your browser and you will see the following EC site.

Similarly, you can access Prometheus at http://localhost:9090 and Jeager at http://localhost:16686.

Datadog Exporter

Now let's send these metrics and traces to Datadog, which is a beta version of the Datadog Exporter for OpenTelemetry collectors, so this is just a trial use.

OpenTelemetry collector Datadog exporter (Datadog document)
https://docs.datadoghq.com/tracing/setup_overview/open_standards/otel_collector_datadog_exporter

Datadog Exporter (Github)
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/datadogexporter

The changes are as follows: Add the Datadog configuration to the Collector configuration file.

Add Datadog settings to exporters. Replace <API_KEY> with your API Key respectively. You can also set the env tag and other tags here. Now it is time to add datadog to trace and metrics in pipelines to be sent. The following configuration will continue to be sent to Prometheus and Jeager.

[src/otelcollector/otelcol-config.yml]

...
exporters:
  jaeger:
    endpoint: "jaeger:14250"
    tls:
      insecure: true
  logging:
  prometheus:
    endpoint: "otelcol:9464"
+ datadog:
+   api:
+     key: <API_KEY>
+   env: otel
+   host_metadata:
+     tags:
+       - otel:true

...
service:
  pipelines:
    traces:
      receivers: [otlp].
      processors: [batch].
- exporters: [logging, jaeger]
+ exporters: [logging, jaeger, datadog]
    metrics:
      receivers: [otlp] processors: [batch
      processors: [batch] + exporters: [prometheus, [batch]]
- exporters: [prometheus, logging]
+ exporters: [prometheus, logging, datadog]

Also, the Datadog Exporter is not included in the container image in the original docker-compose.yaml, so change the image to include the contrib.

[./docker-compose.yaml]

  # Collector
  otelcol:
- image: otel/opentelemetry-collector:0.52.0
+ image: otel/opentelemetry-collector-contrib:0.52.0
    command: [ "--config=/etc/otelcol-config.yml" ]

It shows up in Datadog!

I docker compose up again, accessed the EC site, and the traces and metrics showed up in Datadog as well! I was really impressed with how simple and easy it was.

Service map: I can see the call relationships between microservices and the traffic flow!

Trace: In the individual traces, you can see the backend invocation relationships, the time taken, and other detailed information!

Metrics: Metrics are also sent and tagged!

You can also receive them in Datadog Agent

In this demonstration environment, metrics and traces are received by OpenTelemetry Collector and sent to Datadog, but Datadog Agent can be used instead. If you are already running Datadog Agent, you can use it as is. (However, the Agent must be the latest version.) As shown in the figure below, there are several ways to combine Datadog Agent with OpenTelemetry. Please refer to Datadog's blog post for details.

Ingest OpenTelemetry traces and metrics with the Datadog Agent https://www.datadoghq.com/blog/ingest-opentelemetry-traces-metrics-with-datadog-agent/