SHA256 sha256:cae2bfcec725aaf4930fad73d4e197678382cd8c840021802625be782430c011
Clayton Coleman smarterclayton
smarterclayton
/ metrics_queries
Last active
February 24, 2021 02:08
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| cluster:usage:cpu container_cpu_usage_seconds_total{id="/"} |
smarterclayton
/ gist:b2d16a476434eb818a69d4029181e219
Created
August 10, 2020 18:01
Profiling etcd servers
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| the following command should be run against the most memory intensive DB and the heap uploaded here (SUBSET is pbj3, f3g1, or 428p): | |
| ```gcloud compute ssh --zone us-east1-c --project openshift-ci-infra origin-ci-ig-m-SUBSET --command 'sudo curl --cert /etc/etcd/peer.crt --key /etc/etcd/peer.key --cacert /etc/etcd/ca.crt https://10.142.0.4:2379/debug/pprof/heap' > /tmp/heap``` | |
| If I'm dead, you can view the profile with: | |
| You can view the profile by getting the binary: | |
| ```oc rsh -T -n kube-system master-etcd-origin-ci-ig-m-pbj3 cat /usr/local/bin/etcd > /tmp/etcd``` | |
| and then running the following command to see the profile: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| set -exuo pipefail | |
| # This scripts searches the directories passed as arguments for known failure causes from a set of symptom inputs. | |
| # This is currently experimental and subject to change. | |
| function xmlescape() { | |
| echo -n "$1" | sed 's/&/\&/g; s/</\</g; s/>/\>/g; s/"/\"/g; s/'"'"'/\'/g' | |
| } | |
| tests=$( mktemp -t result-XXXX ) |
smarterclayton
/ gist:fc93e5b3c1007e7d787c3bab0c71793a
Last active
March 29, 2019 15:09
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| {client="openshift-apiserver/v1.12.4+432c257 (linux/amd64) kubernetes/432c257",code="201",contentType="application/vnd.kubernetes.protobuf",endpoint="https",job="apiserver",namespace="default",resource="subjectaccessreviews",scope="cluster",service="kubernetes",verb="POST"} 32.70267684159353 | |
| {client="hyperkube/v1.12.4+432c257 (linux/amd64) kubernetes/432c257/tokens-controller",code="200",contentType="application/vnd.kubernetes.protobuf",endpoint="https",job="apiserver",namespace="default",resource="serviceaccounts",scope="namespace",service="kubernetes",verb="GET"} 9.714850795743688 | |
| {client="openshift-tests/v1.12.4+432c257 (linux/amd64) kubernetes/432c257",code="200",contentType="application/json",endpoint="https",job="apiserver",namespace="default",resource="pods",scope="namespace",service="kubernetes",verb="LIST"} 6.3000083546153745 | |
| {client="openshift-tests/v1.12.4+432c257 (linux/amd64) kubernetes/432c257",code="200",contentType="application/json",endpoint="https",job="apiserver",namespace="default",resourc |
smarterclayton
/ test3.md
Last active
February 13, 2019 04:49
oc adm release info --changelog=/tmp/git registry.svc.ci.openshift.org/openshift/origin-release:{4.0.0-0.alpha-2019-02-11-104425,4.0.0-0.alpha-2019-02-13-001129}
smarterclayton
/ test2.md
Created
February 13, 2019 04:12
smarterclayton
/ test.md
Last active
February 13, 2019 04:08
Annotated history of an upgrade
$ oc patch clusterversion/version --patch '{"spec":{"upstream":"https://origin-release.svc.ci.openshift.org/graph"}}' --type=merge
$ oc adm upgrade --to-latest
# CVO sees new available version, creates a job that can pull the image and extract contents across a hostPath
Dec 27 16:05:05.617 I ns=openshift-cluster-version job=version-4.0.0-0.alpha-2018-12-26-225801-nz5tw Created pod: version-4.0.0-0.alpha-2018-12-26-225801-nz5tw-mnflr
Dec 27 16:05:07.539 I ns=openshift-cluster-version pod=version-4.0.0-0.alpha-2018-12-26-225801-nz5tw-mnflr pulling image "registry.svc.ci.openshift.org/openshift/origin-release:4.0.0-0.alpha-2018-12-26-225801"
Dec 27 16:05:09.688 I ns=openshift-cluster-version pod=version-4.0.0-0.alpha-2018-12-26-225801-nz5tw-mnflr Successfully pulled image "registry.svc.ci.openshift.org/openshift/origin-release:4.0.0-0.alpha-2018-12-26-225801"
smarterclayton
/ gist:053d1e8dd10ff582d1e4ff20d8021ad2
Created
December 27, 2018 18:20
Rough timeline of pod start
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| NAMESPACE NAME READY STATUS RESTARTS AGE | |
| openshift-cluster-kube-apiserver-operator openshift-cluster-kube-apiserver-operator-5c44566d56-hz5s7 1/1 Running 1 33m | |
| openshift-cluster-kube-scheduler-operator openshift-cluster-kube-scheduler-operator-5bfd6877d6-96sk7 1/1 Running 0 33m | |
| openshift-cluster-kube-controller-manager-operator openshift-cluster-kube-controller-manager-operator-6c496f7w5p9r 1/1 Running 0 33m | |
| openshift-cluster-openshift-apiserver-operator openshift-cluster-openshift-apiserver-operator-845b8c9b57-pjxjt 1/1 Running 1 33m | |
| openshift-dns-operator dns-operator-9b99c5887-d2zkd 1/1 Running 0 33m | |
| openshift-core-operators |
smarterclayton
/ gist:d0cf474e67c18435bedba9942aa3ab99
Created
November 26, 2018 20:00
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # HELP etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds Bucketed histogram of db compaction pause duration. | |
| # TYPE etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds histogram | |
| etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="1"} 0 | |
| etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="2"} 0 | |
| etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="4"} 0 | |
| etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="8"} 0 | |
| etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="16"} 0 | |
| etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="32"} 0 | |
| etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="64"} 0 | |
| etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="128"} 0 |
NewerOlder