Observability with Open-Telemetry/Prometheus/Grafana

Instructions-opentelemetry-prometheus-grafana.md

Observability with OpenTelemetry/Prometheus/Grafana

Installation

Install Prometheus and Grafana

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack

Install Cert-Manager (Needed for the OpenTelemetry Operator)

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.yaml

Install OpenTelemetry Operator

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
helm install opentelemetry-operator open-telemetry/opentelemetry-operator

Install OpenTelemetry Collector

kubectl apply -f - <<EOF
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: ot
spec:
  mode: deployment
  ports:
    - name: prometheus
      port: 9464
      targetPort: 9464
      protocol: TCP
  config: |
    receivers:
      jaeger:
        protocols:
          grpc:
      otlp:
        protocols:
          grpc:
          http:

    processors:

    exporters:
      logging:
        verbosity: detailed
      prometheus:
        endpoint: 0.0.0.0:9464
        send_timestamps: true
        enable_open_metrics: true

    service:
      pipelines:
        traces:
          receivers: [ jaeger ]
          processors: []
          exporters: [ logging ]
        metrics:
          receivers: [ otlp ]
          exporters: [ prometheus, logging ]
EOF

Install ServiceMonitor (So Prometheus pulls the data from OpenTelemtry Collector)

kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ot-collector-service-monitor
  labels:
    release: prometheus
spec:
  endpoints:
  - port: prometheus
  selector:
    matchLabels:
      app.kubernetes.io/name: "ot-collector"
EOF

Access

Access Grafana (Address: localhost:9000 / user: admin / password: prom-operator)

kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090

Access Prometheus Dashboard (Address: localhost:9090)

kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090

Access metrics exposed by OpenTelemetry Collector to Prometheus

localhost:9464/metrics