OpenShift Monitoring Development

monitoring-devel.md

Monitoring platform development

To launch a cluster with the the monitoring platform installed, set up a GCE cluster using its instructions, and enable the monitoring platform before bringing up your cluster by adding gcp-dev/myvars.yaml:

openshift_monitoring_deploy: true
# openshift_cluster_monitoring_operator_image: quay.io/dmace/cluster-monitoring-operator:7c7e214

# openshift_cluster_monitoring_operator_alertmanager_config: |+
#   global:
#     resolve_timeout: 5m
#   route:
#     group_wait: 30s
#     group_interval: 5m
#     repeat_interval: 12h
#     receiver: default
#     routes:
#     - match:
#         alertname: DeadMansSwitch
#       repeat_interval: 5m
#       receiver: deadmansswitch
#   receivers:
#   - name: default
#   - name: deadmansswitch

# openshift_monitor_availability_install: true
# openshift_monitor_app_create_image: quay.io/dmace/openshift-monitor-project-lifecycle:0d619bf
# openshift_monitor_app_create_run_interval: 30s
# openshift_monitor_app_create_log_level: 2

To log into the Prometheus or Alertmanager consoles, you need a user with some broad permissions:

oc adm policy add-cluster-role-to-user cluster-reader dmace

The consoles are here (WHAT is the prefix you used in the GCP make up command):

https://prometheus-k8s-openshift-monitoring.apps.$WHAT.origin-gce.dev.openshift.com

https://alertmanager-main-openshift-monitoring.apps.$WHAT.origin-gce.dev.openshift.com

Testing role/playbook updates

Make changes in openshift-ansible, and create a new image:

docker build -f images/installer/Dockerfile -t ironcladlou/openshift-ansible:latest .

Update any variables you want in gcp-dev/vars.yaml, then run the ad-hoc playbooks:

$ make WHAT=dmace OPENSHIFT_ANSIBLE_IMAGE=ironcladlou/openshift-ansible:latest sh
# Now, inside the container...
$ ansible-playbook playbooks/openshift-monitoring/install-gcp.yml

Cluster Monitoring Operator

It's a Kube deployment:

oc get -n openshift-monitoring deployments/cluster-monitoring-operator

To deploy new images quickly, just patch the deployment.

Testing a new ServiceMonitor

If it's new and not included in the image, just use:

oc apply -n openshift-monitoring -f /some/servicemonitor.yaml

If modifying one included in the image, direct edits may be overwritten by the Cluster Monitoring Operator, so deploy a new image to make changes persistent.

Testing new rules

Deploy a new Cluster Monitoring Operator image.

Set up a GCE cluster with the monitoring platform enabled ...

Do I get it right that I need GCE account? I do not think I have any. Could I use AWS instead?

To work in GCP you need access to https://github.com/openshift/shared-secrets, but it's recommended you get GCP access so you can mess with things in the web console too. I don't have an answer for AWS right now- it's possible with openshift-ansible, but I don't have a simple set of instructions for it and afaik the test-deploy stuff doesn't yet work with AWS.

I have access to that repo, but still I think I need to ask for GCP account. I have access to AWS account that I could use but if there are no scripts for this that will help me deploy tests then I think I need to go GCP way.

I don't see anything specific to gcp or aws in the above instructions.

@richm, the test-deploy tooling is GCP-centric (although @smarterclayton has mentioned AWS support is desired)

I created ticket for the GCE access and I will try to see if I can get this working on AWS in the meantime.
(Agh! I will need to publish my new GPG first)