Skip to content

Instantly share code, notes, and snippets.

@cmwylie19

cmwylie19/Prometheus-Operator.md

Last active May 16, 2022 12:01
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save cmwylie19/8493cfd7733fdc7beeeb7e7067968e5e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save cmwylie19/8493cfd7733fdc7beeeb7e7067968e5e to your computer and use it in GitHub Desktop.
Download ZIP
Prometheus-Operator
Raw
Prometheus-Operator.md

Prometheus Operator

To install, create the namespace, operatorgroup, and subscription.

oc apply -f -<<EOF
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: prometheus
spec: {}
status: {}
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: prometheus-j6qbp
  namespace: prometheus
spec:
  targetNamespaces:
  - prometheus
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: prometheus
  namespace: prometheus
spec:
  channel: beta
  installPlanApproval: Automatic
  name: prometheus
  source: community-operators
  sourceNamespace: openshift-marketplace
  startingCSV: prometheusoperator.0.47.0
EOF

Next, spin up an instance of the Prometheus Operator, with a ConfigMap for the Client ID and a secret for the Client Secret

kubectl apply -f -<<EOF
apiVersion: v1
data:
  client-secret: asd
kind: Secret
metadata:
  creationTimestamp: null
  name: obs-creds
  namespace: prometheus
---
apiVersion: v1
data:
  client-id: asd
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: obs-credentials
  namespace: prometheus
---
kind: Prometheus
apiVersion: monitoring.coreos.com/v1
metadata:
  name: blue-prom
  namespace: prometheus
spec:
  serviceMonitorSelector: {}
  serviceMonitorNamespaceSelector: {}
  logLevel: debug
  replicas: 1
  image: quay.io/prometheus/prometheus:v2.30.0
  serviceAccountName: prometheus-k8s
  remoteWrite:
  - url: 'https://url.com/api/metrics/v1/test/api/v1/receive'
    tlsConfig:
      insecureSkipVerify: true
    oauth2:
     clientId:
       configMap:
         key: client-id
         name: obs-credential
     clientSecret:
       key: client-secret
       name: obs-creds
     tokenUrl: 'https://url.com/dex/token'
  resources:
    requests:
      memory: 400Mi 
EOF

Deploy Blue App which produces metrics:

kubectl apply -f -<<EOF
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: null
  name: blue
spec: {}
status: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: blue
    version: v1
  name: blue
  namespace: blue
spec:
  ports:
    - port: 9000
      name: http
  selector:
    app: blue
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: blue
    version: v1
  name: blue
  namespace: blue
spec:
  selector:
    matchLabels:
      app: blue
      version: v1
  replicas: 1
  template:
    metadata:
      labels:
        app: blue
        version: v1
    spec:
      serviceAccountName: blue
      containers:
        - image: docker.io/cmwylie19/metrics-demo
          name: blue
          resources:
            requests:
              memory: "64Mi"
              cpu: "250m"
            limits:
              memory: "128Mi"
              cpu: "500m"
          ports:
            - containerPort: 9000
              name: http
          imagePullPolicy: Always
      restartPolicy: Always
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: blue
  namespace: blue
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  creationTimestamp: null
  labels:
    app: blue
    version: v1
  name: blue
  namespace: blue
spec:
  port:
    targetPort: http
  to:
    kind: ""
    name: blue
    weight: null
EOF

ServiceMonitor for Blue app in Blue Namespace

kubectl apply -f -<<EOF
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: blue
  name: blue
  namespace: prometheus
spec:
  endpoints:
  - port: http
  namespaceSelector:
    matchNames:
    - blue
  selector:
    matchLabels:
      app: blue
EOF

Get logs of the blue prometheus instance

k logs prometheus-blue-prom-0 -n prometheus -c prometheus

output:

level=debug ts=2022-05-16T11:46:20.545Z caller=klog.go:55 component=k8s_client_runtime func=Verbose.Infof msg="Listing and watching *v1.Pod from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167"
level=debug ts=2022-05-16T11:46:20.550Z caller=klog.go:84 component=k8s_client_runtime func=Infof msg="GET https://172.30.0.1:443/api/v1/namespaces/blue/pods?limit=500&resourceVersion=0 403 Forbidden in 5 milliseconds"
level=error ts=2022-05-16T11:46:20.550Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:prometheus:prometheus-k8s\" cannot list resource \"pods\" in API group \"\" in the namespace \"blue\""
level=debug ts=2022-05-16T11:46:24.034Z caller=klog.go:55 component=k8s_client_runtime func=Verbose.Infof msg="Listing and watching *v1.Service from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167"
level=debug ts=2022-05-16T11:46:24.035Z caller=klog.go:84 component=k8s_client_runtime func=Infof msg="GET https://172.30.0.1:443/api/v1/namespaces/blue/services?limit=500&resourceVersion=0 403 Forbidden in 1 milliseconds"
level=error ts=2022-05-16T11:46:24.035Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:prometheus:prometheus-k8s\" cannot list resource \"services\" in API group \"\" in the namespace \"blue\""
level=debug ts=2022-05-16T11:46:26.627Z caller=klog.go:55 component=k8s_client_runtime func=Verbose.Infof msg="Listing and watching *v1.Endpoints from pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167"
level=debug ts=2022-05-16T11:46:26.628Z caller=klog.go:84 component=k8s_client_runtime func=Infof msg="GET https://172.30.0.1:443/api/v1/namespaces/blue/endpoints?limit=500&resourceVersion=0 403 Forbidden in 1 milliseconds"
level=error ts=2022-05-16T11:46:26.629Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:prometheus:prometheus-k8s\" cannot list resource \"endpoints\" in API group \"\" in the namespace \"blue\""

This means that the prometheus/prometheus-k8s service account can scrape pods,services, and endpoints:

k auth can-i get services -n blue --as system:serviceaccount:prometheus:prometheus-k8s
k auth can-i get endpoints -n blue --as system:serviceaccount:prometheus:prometheus-k8s
k auth can-i get pods -n blue --as system:serviceaccount:prometheus:prometheus-k8s

Create Cluster Role/Binding so that prometheus/prometheus-k8s service account can scrape pods,services, and endpoints:

kubectl apply -f -<<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: blue-resources
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - pods/status
  - endpoints
  - services
  verbs:
  - list
  - get
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: null
  name: blue-resources-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: blue-resources
subjects:
- kind: ServiceAccount
  name: prometheus-k8s
  namespace: prometheus
EOF

Test that the ClusterRole and Binding will allow the service account to scrape the resource

k auth can-i get services -n blue --as system:serviceaccount:prometheus:prometheus-k8s 
k auth can-i get endpoints -n blue --as system:serviceaccount:prometheus:prometheus-k8s 
k auth can-i get pods -n blue --as system:serviceaccount:prometheus:prometheus-k8s

you should now see the blue servicemonitor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment