otel prometheus to splunk hec

prometheus-hec.md

scraping prometheus endpoints with otel

the otel collector is a reasonably nice way to scrape prometheus enpoints

it is obviously designed for use with splunk's olly cloud, but the otel collector supports output to splunk hec tokens as well.

im running multiple docker containers (to seperate workloads).

first, create configs

# no extensions, we're trying to keep things as lean as possbile
extensions:

receivers:
  smartagent/prometheus-exporter1:
    - type: prometheus-exporter
      host: app1-url.com.au
      metricPath: /private/actuator/prometheus
      port: 443
      useHTTPS: true
      skipVerify: true
      extraDimensions:
        metric_source: prometheus
        prometheus_job: prom_job_1
  smartagent/prometheus-exporter2:
    - type: prometheus-exporter
      host: app2-url.com.au
      metricPath: /private/actuator/prometheus
      port: 443
      useHTTPS: true
      skipVerify: true
      extraDimensions:
        metric_source: prometheus
        prometheus_job: prom_job_2
  smartagent/prometheus-exporter3:
    - type: prometheus-exporter
      host: app3-url.com.au
      metricPath: /private/actuator/prometheus
      port: 443
      useHTTPS: true
      skipVerify: true
      extraDimensions:
        metric_source: prometheus
        prometheus_job: prom_job_3

processors:
  batch:
  memory_limiter:
    check_interval: 2s
    limit_mib: 128

exporters:
  splunk_hec:
    token: "hec-token-hec-token-hec-token-hec-to"
    endpoint: "https://your-splunk-url:8088/services/collector"
    source: "otel"
    sourcetype: "otel"
    index: "otel_metrics"
    tls:
      insecure_skip_verify: true
    splunk_app_name: "OpenTelemetry-Collector Splunk Exporter"

service:
  # extensions: [health_check, http_forwarder, zpages, memory_ballast]
  pipelines:
    metrics/prometheus:
      receivers:
        - smartagent/prometheus-exporter1
        - smartagent/prometheus-exporter2
        - smartagent/prometheus-exporter3
      processors: [memory_limiter, batch]
      exporters: [splunk_hec]

if you are running olly from docker you'll need to give it access to the docker sock to capture data

setfacl -m u:splunk-otel-collector:rw /var/run/docker.sock

then you can debug (logs to stdout) with the following command

docker run \
    --rm \
    --env SPLUNK_CONFIG=/etc/collector.yaml \
    --volume "${PWD}/otel-clapi.yaml":/etc/collector.yaml:ro \
    --name otel-clapi \
    quay.io/signalfx/splunk-otel-collector:latest

or run it "properly" using the following

docker run \
    --detach \
    --restart=always \
    -e SPLUNK_CONFIG=/etc/collector.yaml \
    -v "${PWD}/otel-workload1.yaml":/etc/collector.yaml:ro \
    --name otel-workload1 \
    quay.io/signalfx/splunk-otel-collector:latest