Add additinal scrape configs to Prometheus k8s

This procedure takes into account Prometheus Operator has been installed using the manifests provided in prometheus-operator/contrib/kube-prometheus/manifests/ on https://github.com/coreos/prometheus-operator.git

This procedure is based on this documentation

The idea behind this is to fetch the metrics through the apiserver instead of fetching kubelet directly.

Allow additional scrape configs in Prometheus

Edit the file prometheus-operator/contrib/kube-prometheus/manifests/prometheus-prometheus.yaml

In the spec section, add the following:

spec:
  additionalScrapeConfigs:
    name: additional-scrape-configs
    key: prometheus-additional-sc.yaml

Apply new configuration

$ kubectl apply -f prometheus-operator/contrib/kube-prometheus/manifests/prometheus-prometheus.yaml
prometheus.monitoring.coreos.com/k8s configured

Manage permissions

The serviceaccount running the Prometheus pods mu be allowed to get information about nodes (see next section)

Create a file prometheus-cr-crb.yaml:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-clusterrole
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  verbs: ["watch", "get", "list"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus-clusterrole-binding
roleRef:
  kind: ClusterRole
  name: prometheus-clusterrole
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: prometheus-k8s
  namespace: monitoring

Create new permissions:

$ kubectl create -f prometheus-cr-crb.yaml
clusterrole.rbac.authorization.k8s.io/prometheus-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-clusterrole-binding created

Create scrape configuration secret

Create a file prometheus-additional-sc.yaml with the following:

- job_name: "kubernetes-nodes-cadvisor"
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
    - role: node
  relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: kubernetes.default.svc:443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
- job_name: "kubernetes-nodes"
  scheme: https
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    insecure_skip_verify: true
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
    - role: node
  relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: kubernetes.default.svc:443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/$1/proxy/metrics

Generate the secret from the file, the filename is very important as it will be the key used by Prometheus to get the additional scrapes:

$ kubectl create -n monitoring secret generic additional-scrape-configs --from-file prometheus-additional-sc.yaml 
secret/additional-scrape-configs created

Validation

In the targets http://PROMETHEUS_URL/targets should appear the scrapes for:

kubernetes-nodes
kubernetes-nodes-cadvisor

Troubleshoot

If the targets don't show up quickly enought and the scrape configuration is not present in http://PROMETHEUS_URL/config we may have to delete the pods:

kubectl -n monitoring delete po prometheus-k8s-0 prometheus-k8s-1