prometheus-operator.md

Prometheus-Operator

Pre-requisites

Install Helm

HELM_VER=v2.8.2
wget -q https://kubernetes-helm.storage.googleapis.com/helm-${HELM_VER}-linux-amd64.tar.gz
tar -zxvf helm-${HELM_VER}-linux-amd64.tar.gz
sudo mv linux-amd64/helm /usr/local/bin/helm

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
EOF

helm init --wait --service-account tiller

Prometheus

Install and Setup Prometheus-Operator

helm repo add coreos \
https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/

helm install coreos/prometheus-operator \
--name prometheus-operator --namespace monitoring

helm install coreos/kube-prometheus \
--name kube-prometheus --namespace monitoring

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: default
EOF

Usage

Spin up dedicated Prometheus instance for example-app deployment

cat << EOF | kubectl apply -f -
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: example-app
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
      - name: example-app
        image: fabxc/instrumented_app
        ports:
        - name: web
          containerPort: 8080
---
kind: Service
apiVersion: v1
metadata:
  name: example-app
  labels:
    app: example-app
spec:
  selector:
    app: example-app
  ports:
  - name: web
    port: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  labels:
    team: frontend
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: web
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: example-app
spec:
  serviceAccountName: prometheus
  serviceMonitorSelector:
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi
  storage:
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
  name: example-app-prometheus
spec:
  ports:
  - name: web
    port: 9090
    protocol: TCP
    targetPort: web
  selector:
    prometheus: example-app
EOF

Interact

Query the Prometheus server for metrics from example-app. Grafana is populated with off-the-shelf dashboards for Kubernetes monitoring.

kubectl port-forward prometheus-example-app-0 9090 &

GRAF_POD=$(kubectl get pods -n monitoring -l app=kube-prometheus-grafana \
--output=jsonpath="{.items..metadata.name}")
kubectl port-forward ${GRAF_POD} 3000 -n monitoring &