opentelemetry-collector and VM

README.md

How to use opentelemetry collector with VictoriaMetrics

requirements

• helm
• kubectl
• kubernetes cluster

Installation of VictoriaMetrics

In education purpose, I'll use single node version of VictoriaMetrics.

Add helm repository to your local PC:

helm repo add vm https://victoriametrics.github.io/helm-charts/
helm repo update

Install single node version:

helm install victoria-metrics vm/victoria-metrics-single

Check it's up and running:

kubectl get pods
# victoria-metrics-victoria-metrics-single-server-0   1/1     Running   0          3m1s

Helm chart provides following urls for reading and writing data:

  Write url inside the kubernetes cluster:
   http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write

Read Data:
 The following url can be used as the datasource url in Grafana::
   http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428

setup opentelemetry collector

Official opentelemetry collector docs have various installation methods. I'll use simple kubernetes yaml manifest.

It consists of three parts - configmap, deployment and service.

Let's quickly check our configuration, it's pretty simple:

    # supported ingestion methods https://opentelemetry.io/docs/collector/configuration/#receivers
    # for our case only otlp is needed
    receivers:
      otlp:
        protocols: 
          http:
          grpc:
    # there is no need in any processor, but it must be defined.
    processors:
    # as exporter we'll use VictoriaMetrics write endpoint from helm chart output.
    exporters:
      prometheusremotewrite:
        endpoint: "http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write"
    service:
      # group up pipeline
      pipelines:
        metrics:
          receivers: [otlp]
          processors: []
          exporters: [prometheusremotewrite]

Full version of possible configuration options could be found at docs

Apply kubernetes manifest from collector.yaml file.

kubectl apply -f https://gist.githubusercontent.com/f41gh7/78d8f01de0ae0a557ed638dff775a97d/raw/4f42f4bcbf7273858e76f40362efe48db439bf94/collector.yaml

Optional install collector agent

Install opentelemetry agent for collecting information from hosts and forwarding it into opentelemetry collector:

kubectl apply -f https://gist.githubusercontent.com/f41gh7/78d8f01de0ae0a557ed638dff775a97d/raw/4f42f4bcbf7273858e76f40362efe48db439bf94/agent.yaml

Check collected metrics with port-forwarding VictoriaMetrics kubectl port-forward service/victoria-metrics-victoria-metrics-single-server 8428 at webui

agent.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-agent-conf
  labels:
    app: opentelemetry
    component: otel-agent-conf
data:
  otel-agent-config: |
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: otel-collector
              scrape_interval: 5s
              static_configs:
                - targets: [localhost:8888]
      hostmetrics:
        scrapers:
          cpu:
          disk:
          filesystem:
          load:
          memory:
          network:
          process:
          processes:
          paging:
    exporters:
      otlp:
        endpoint: "oltp-collector.default.svc.cluster.local:4317"
        tls:
          insecure: true
        sending_queue:
          num_consumers: 4
          queue_size: 100
        retry_on_failure:
          enabled: true
    processors:
    service:
      pipelines:
        metrics:
          receivers: [prometheus,hostmetrics]
          processors: []
          exporters: [otlp]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-agent
  labels:
    app: opentelemetry
    component: otel-agent
spec:
  selector:
    matchLabels:
      app: opentelemetry
      component: otel-agent
  template:
    metadata:
      labels:
        app: opentelemetry
        component: otel-agent
    spec:
      containers:
      - command:
          - "/otelcol"
          - "--config=/conf/otel-agent-config.yaml"
        image: otel/opentelemetry-collector:0.74.0
        name: otel-agent
        resources:
          limits:
            cpu: 500m
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 100Mi
        ports:
        - containerPort: 55679 # ZPages endpoint.
        - containerPort: 4317 # Default OpenTelemetry receiver port.
        - containerPort: 8888  # Metrics.
        volumeMounts:
        - name: otel-agent-config-vol
          mountPath: /conf
      volumes:
        - configMap:
            name: otel-agent-conf
            items:
              - key: otel-agent-config
                path: otel-agent-config.yaml
          name: otel-agent-config-vol

collector.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: otlp-collector-config
data:
  collector.yaml: |
    receivers:
      otlp:
        protocols: 
          http:
          grpc:
    # leave it empty
    processors:
    exporters:
      prometheusremotewrite:
        endpoint: "http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write"
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: []
          exporters: [prometheusremotewrite]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/component: otlp-collector
  name: oltp-collector
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: otlp-collector
  template:
    metadata:
      labels:
        app.kubernetes.io/component: otlp-collector
    spec:
      containers:
        - args:
            - --config
            - /opt/otlp-collector-config/collector.yaml
          image: otel/opentelemetry-collector
          imagePullPolicy: IfNotPresent
          name: collector
          ports:
            - containerPort: 4318
              name: http-otlp
              protocol: TCP
            - containerPort: 4317
              name: grpc-otlp
              protocol: TCP
            - containerPort: 8888
              name: metrics
              protocol: TCP
          resources:
            limits:
              cpu: 250m
              memory: 1Gi
            requests:
              cpu: 50m
              memory: 151Mi
          volumeMounts:
            - mountPath: /opt/otlp-collector-config
              name: otlp-collector-config
      volumes:
        - configMap:
            defaultMode: 420
            name: otlp-collector-config
          name: otlp-collector-config
---
apiVersion: v1
kind: Service
metadata:
  name: oltp-collector
spec:
  ports:
    - name: http-otlp
      port: 4318
      protocol: TCP
      targetPort: 4318
    - name: grpc-otlp
      port: 4317
      protocol: TCP
      targetPort: 4317
  selector:
    app.kubernetes.io/component: otlp-collector
  type: ClusterIP

@f41gh7 thanks for this

    exporters:
      prometheusremotewrite:
        endpoint: "http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write"

now that vm supports otlp, shouldn't that be used?

https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#sending-data-via-opentelemetry

@f41gh7 Thanks for this gist!
It is worth mentioning the case in which it is a cluster of VM. The endpoint is this case would be different. In my case, I used:

  exporters:
      prometheusremotewrite:
        endpoint: "http://vmcluster-vminsert.monitoring.svc.cluster.local:8480/insert/multitenant/prometheus/api/v1/write"

Reference
https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#url-format

@f41gh7 thanks for this

    exporters:
      prometheusremotewrite:
        endpoint: "http://victoria-metrics-victoria-metrics-single-server.default.svc.cluster.local:8428/api/v1/write"

now that vm supports otlp, shouldn't that be used?

https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#sending-data-via-opentelemetry

Yes, it's defensively possible to use direct push to the vmagent/vminsert/vmsingle.

I have an example client for it at the following gist

https://gist.github.com/f41gh7/1a67850e2cc795af05d96085de9c6653

Thanks for this solution! Your teaching is really important to me. I have been looking for help on the Internet for a long time. Tried using https://essays.edubirdie.com/homework-help to do the job. Usually it was university homework and they were great at it, but I wanted to find something more themed. I recently came across this site and I really like it. I have already solved some problems with its help.