Setup Prometheus to scrape external node-exporter

_Prometheus scrape external endpoints.md

Setup Prometheus to scrape external node-exporter

This came out of an ansible role I was creating to monitor some external hosts that run node-exporter. I've since moved away from exposting node-exporter and am using prometheus pushgateway instead. This gist exists to capture what I learned.

How does it work?

Deploy the endpoint, service, and servicemonitor in the openshift-monitoring namespace. You should be able to deploy to another project but you'll maybe have problems with reading data. A couple of notes on that at the end

Ansible snippet to make this happen (I have not tested this). Assumes the host it runs on has ability to run oc commands against openshift-monitoring.

- set_fact:
    g_name: my-custom-endpoints
    g_namespace: openshift-monitoring
    g_port: 9100
    g_ip_addresses:
      - 10.0.1.123
      - 10.0.2.234

- name: "Stage files"
  template:
    src: "{{ item }}.j2"
    dest: "/tmp/{{ item }}"
    mode: 0600
  with_items:
    - endpoint.yaml
    - service.yaml
    - servicemonitor.yaml

- name: "Apply stuff"
  shell: "oc apply -f /tmp/{{ item }}"
  with_items:
    - endpoint.yaml
    - service.yaml
    - servicemonitor.yaml

Deploying to another namespace?

Few things to look at, not a detailed how to:

• grant cluster-reader to prometheus-k8s
• deploy servicemonitor in openshift-monitoring namespace
• make sure servicemonitor has a namespace selector

endpoint.yaml.j2

# vim: expandtab:tabstop=2:shiftwidth=2:autoindent
---
apiVersion: v1
kind: Endpoints
metadata:
    name: {{ g_name }}
    namespace: {{ g_namespace }}
    labels:
        k8s-app: {{ g_name }}
subsets:
    - addresses:
{% for ip in g_ip_addresses %}
      - ip: {{ ip }}
{% endfor %}
      ports:
        - name: metrics
          port: {{ g_port }}

service.yaml.j2

# vim: expandtab:tabstop=2:shiftwidth=2:autoindent
---
apiVersion: v1
kind: Service
metadata:
    name: {{ g_name }}
    namespace: {{ g_namespace }}
    labels:
        k8s-app: {{ g_name }}
spec:
    ports:
    - name: metrics
      port: {{ g_port }}
      protocol: TCP
      targetPort: {{ g_port }}
    selector: {}

servicemonitor.yaml.j2

# vim: expandtab:tabstop=2:shiftwidth=2:autoindent
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
    name: {{ g_name }}
    namespace: {{ g_namespace }}
    labels:
        k8s-app: {{ g_name }}
spec:
    selector:
        matchLabels:
            k8s-app: {{ g_name }}
    endpoints:
    - port: metrics
      interval: 10s
      honorLabels: true