jewzaam

Setup Prometheus to scrape external node-exporter

This came out of an ansible role I was creating to monitor some external hosts that run node-exporter. I've since moved away from exposting node-exporter and am using prometheus pushgateway instead. This gist exists to capture what I learned.

How does it work?

Deploy the endpoint, service, and servicemonitor in the openshift-monitoring namespace. You should be able to deploy to another project but you'll maybe have problems with reading data. A couple of notes on that at the end

Ansible snippet to make this happen (I have not tested this). Assumes the host it runs on has ability to run oc commands against openshift-monitoring.

jewzaam

https://github.com/openshift/hive

# login to the target cluster with oc
go get sigs.k8s.io/kustomize
go get sigs.k8s.io/controller-tools
go get github.com/openshift/hive
cd $GOPATH/src/github.com/openshift/hive
make deploy

jewzaam

#!/bin/bash

# /api
GROUP=""
for VERSION in `oc get --raw /api | jq -r .versions[] | sort`;
do
    echo "- apiGroups:"
    echo "  - $GROUP"

    for RESOURCE in `oc get --raw /api/${VERSION} | jq -r .resources[].name | sort`;

jewzaam

Assumes you have an OCP 4.x cluster already created and you are logged in.

A few gotyas:
* Do not use spaces in your callback name.  In this example it's called "github".
* If you want to use a team, the format in the config (not shown here) is "<ORGANIZATION_NAME>/<TEAM_NAME>"

# Create github oauth app with:

# hostname
echo "https://`oc get routes -n openshift-console | grep -v NAME | awk '{print $2}'`"

jewzaam

6.1. GPG (PGP)

As an ops person, you need a GPG key associated with your @redhat.com email address and distributed to a public keyserver such as pgp.mit.edu.

More information on specific steps can currently be found starting here, or via a web search: https://www.gnupg.org/gph/en/manual/c14.html

If you are creating a new GPG key, make sure it's 4096 bits long. If you're sure you want to re-use an existing key, you may do so. 4096 bits is preferred,

jewzaam

#!/bin/bash

# /api
GROUP=""
for VERSION in `oc get --raw /api | jq -r .versions[] | sort`;
do
    echo "- apiGroups:"
    echo "  - $GROUP"

    for RESOURCE in `oc get --raw /api/${VERSION} | jq -r .resources[].name | sort`;

jewzaam

# Useful link for how to discover API's: https://github.com/operator-framework/operator-registry/issues/33#issuecomment-473671805

# get tools
go get github.com/fullstorydev/grpcurl/...
go install github.com/fullstorydev/grpcurl/cmd/grpcurl

# constants for what we're investigating
NAMESPACE=openshift-marketplace
LOCAL_PORT=50051
REMOTE_PORT=50051

jewzaam

import re

from kubernetes import client, config
from kubernetes.client import ApiClient

# based on initial 4.3.0 installation using 3 tiers of crd name possible:
# oc get crds -o json | jq -r '.items[].metadata.name' | sed 's/.*\(\.[^.]*\.[^.]*\.[^.]*\)$/\1/g' | sed 's/^[^.]*\(\.[^.]*\.[^.]*\)/\1/g' | sort -u

RE_DENYLIST = [
    r".*\.authorization\.openshift\.io",

jewzaam

python -c 'import json, sys, yaml ; y=yaml.safe_load(sys.stdin.read()) ; print(json.dumps(y))'

jewzaam

global:
  resolve_timeout: 5m
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
route:
  receiver: "null"
  group_by:
  - job
  routes:
  - receiver: "null"
    group_by: