Skip to content

Instantly share code, notes, and snippets.

@deads2k

deads2k/console access

Last active January 25, 2024 17:05
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save deads2k/2b72a563a3525a61fb326f1a42ca450e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save deads2k/2b72a563a3525a61fb326f1a42ca450e to your computer and use it in GitHub Desktop.
Download ZIP
Raw
console access
from https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.16-e2e-aws-sdn-serial/1750498185056358400/artifacts/e2e-aws-sdn-serial/gather-audit-logs/artifacts/
deads@fedora:~/workspaces/cluster-debug-tools/src/github.com/openshift/cluster-debug-tools$ ./kubectl-dev_tool audit -f '/home/deads/Downloads/audit-logs(4)/9e67a938c529c2e045643d88bb48f105d0d45369e4e535d8e3b1163a47e22dca/audit_logs/kube-apiserver' -otop --by=verb --user=system:serviceaccount:openshift-console-operator:console-operator
had 8600 line read failures
count: 36796, first: 2024-01-25T08:02:24-05:00, last: 2024-01-25T10:12:01-05:00, duration: 2h9m36.97217s
Top 10 "CREATE" (of 41 total hits):
23x [ 10.92813ms] [409-22] /apis/console.openshift.io/v1/consolenotifications [system:serviceaccount:openshift-console-operator:console-operator]
5x [ 6.9752ms] [201-4] /api/v1/namespaces/openshift-console/configmaps [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 11.18ms] [201-1] /apis/console.openshift.io/v1/consoleclidownloads [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 10.2505ms] [201-1] /api/v1/namespaces/openshift-console/services [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 9.2625ms] [201-1] /apis/route.openshift.io/v1/namespaces/openshift-console/routes [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 8.8815ms] [201-1] /apis/policy/v1/namespaces/openshift-console/poddisruptionbudgets [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 29.739ms] [201-1] /apis/apps/v1/namespaces/openshift-console/deployments [system:serviceaccount:openshift-console-operator:console-operator]
1x [ 5.197ms] [] /api/v1/namespaces/openshift-config-managed/configmaps [system:serviceaccount:openshift-console-operator:console-operator]
1x [ 5.99ms] [] /api/v1/namespaces/openshift-console/secrets [system:serviceaccount:openshift-console-operator:console-operator]
1x [ 5.066ms] [] /apis/coordination.k8s.io/v1/namespaces/openshift-console-operator/leases?timeout=1m47s [system:serviceaccount:openshift-console-operator:console-operator]
Top 10 "DELETE" (of 703 total hits):
197x [ 6.63201ms] [404-196] /apis/route.openshift.io/v1/namespaces/openshift-console/routes/console-custom [system:serviceaccount:openshift-console-operator:console-operator]
195x [ 6.876682ms] [404-194] /apis/route.openshift.io/v1/namespaces/openshift-console/routes/downloads-custom [system:serviceaccount:openshift-console-operator:console-operator]
177x [ 2.926209ms] [404-176] /api/v1/namespaces/openshift-console/services/console-redirect [system:serviceaccount:openshift-console-operator:console-operator]
134x [ 3.182268ms] [404-133] /apis/console.openshift.io/v1/consolenotifications/cluster-upgrade [system:serviceaccount:openshift-console-operator:console-operator]
Top 10 "GET" (of 29520 total hits):
9838x [ 3.012615ms] [200-9836,404-1] /apis/apps/v1/namespaces/openshift-console/deployments/console [system:serviceaccount:openshift-console-operator:console-operator]
4919x [ 3.116522ms] [200-4918] /api/v1/namespaces/openshift-console/configmaps/console-config [system:serviceaccount:openshift-console-operator:console-operator]
4919x [ 2.509514ms] [200-4918] /api/v1/namespaces/openshift-config-managed/configmaps/console-public [system:serviceaccount:openshift-console-operator:console-operator]
4919x [ 3.620538ms] [200-4918] /apis/operators.coreos.com/v1/olmconfigs/cluster [system:serviceaccount:openshift-console-operator:console-operator]
1245x [ 3.625317ms] [200-1244] /apis/config.openshift.io/v1/ingresses/cluster [system:serviceaccount:openshift-console-operator:console-operator]
502x [ 5.679274ms] [200-501] /apis/route.openshift.io/v1/namespaces/openshift-console/routes/console [system:serviceaccount:openshift-console-operator:console-operator]
391x [ 6.393424ms] [200-389,404-1] /apis/route.openshift.io/v1/namespaces/openshift-console/routes/downloads [system:serviceaccount:openshift-console-operator:console-operator]
341x [ 7.272217ms] [200-340] /api/v1/namespaces/openshift-console/configmaps/trusted-ca-bundle [system:serviceaccount:openshift-console-operator:console-operator]
340x [ 2.101597ms] [200-339] /api/v1/namespaces/openshift-console/configmaps/default-ingress-cert [system:serviceaccount:openshift-console-operator:console-operator]
306x [ 2.543382ms] [200-305] /apis/config.openshift.io/v1/infrastructures/cluster [system:serviceaccount:openshift-console-operator:console-operator]
Top 10 "LIST" (of 5545 total hits):
4920x [ 6.518282ms] [200-4919] /api/v1/nodes?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
593x [ 5.237092ms] [200-592] /apis/console.openshift.io/v1/consoleplugins?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
3x [ 4.436ms] [200-2] /apis/operators.coreos.com/v1/olmconfigs [system:serviceaccount:openshift-console-operator:console-operator]
3x [ 599.666µs] [200-2] /api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 69.903ms] [200-1] /apis/config.openshift.io/v1/clusterversions?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 3.1585ms] [200-1] /api/v1/namespaces/openshift-console/configmaps?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 15.2205ms] [200-1] /api/v1/namespaces/openshift-config-managed/configmaps?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 1.784ms] [200-1] /api/v1/namespaces/openshift-config/configmaps?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 9.334ms] [200-1] /apis/console.openshift.io/v1/consoleclidownloads?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
1x [ 395µs] [] /apis/apps/v1/namespaces/openshift-console/deployments?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
Top 10 "UPDATE" (of 402 total hits):
300x [ 3.86672ms] [200-299] /apis/coordination.k8s.io/v1/namespaces/openshift-console-operator/leases/console-operator-lock?timeout=1m47s [system:serviceaccount:openshift-console-operator:console-operator]
73x [ 11.505465ms] [200-29,409-43] /apis/operator.openshift.io/v1/consoles/cluster/status [system:serviceaccount:openshift-console-operator:console-operator]
24x [ 9.880333ms] [200-18,409-5] /apis/config.openshift.io/v1/clusteroperators/console/status [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 15.0065ms] [200-1] /apis/apps/v1/namespaces/openshift-console/deployments/console [system:serviceaccount:openshift-console-operator:console-operator]
1x [ 8.476ms] [] /api/v1/namespaces/openshift-console/configmaps/console-config [system:serviceaccount:openshift-console-operator:console-operator]
1x [ 9.919ms] [] /apis/config.openshift.io/v1/consoles/cluster/status [system:serviceaccount:openshift-console-operator:console-operator]
1x [ 13.169ms] [] /apis/oauth.openshift.io/v1/oauthclients/console [system:serviceaccount:openshift-console-operator:console-operator]
Top 10 "WATCH" (of 585 total hits):
59x [6m16.377773881s] [200-58] /api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=22261&timeout=8m1s&timeoutSeconds=481&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
41x [6m13.632001292s] [200-40] /api/v1/namespaces/openshift-config/configmaps?allowWatchBookmarks=true&resourceVersion=22261&timeout=8m5s&timeoutSeconds=485&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
41x [6m12.969064243s] [200-40] /api/v1/namespaces/openshift-config-managed/configmaps?allowWatchBookmarks=true&resourceVersion=22261&timeout=7m33s&timeoutSeconds=453&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
40x [6m10.923063725s] [200-39] /api/v1/namespaces/openshift-console/configmaps?allowWatchBookmarks=true&resourceVersion=22261&timeout=9m18s&timeoutSeconds=558&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
22x [5m50.274165954s] [200-21] /apis/config.openshift.io/v1/authentications?allowWatchBookmarks=true&resourceVersion=22311&timeout=8m4s&timeoutSeconds=484&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
21x [6m8.954359761s] [200-20] /apis/apps/v1/namespaces/openshift-console/deployments?allowWatchBookmarks=true&resourceVersion=22190&timeout=9m15s&timeoutSeconds=555&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
21x [5m54.577417142s] [200-20] /api/v1/namespaces/openshift-console/secrets?allowWatchBookmarks=true&resourceVersion=22282&timeout=9m33s&timeoutSeconds=573&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
21x [6m8.571611333s] [200-20] /apis/config.openshift.io/v1/oauths?allowWatchBookmarks=true&resourceVersion=22307&timeout=8m21s&timeoutSeconds=501&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
21x [6m6.07856519s] [200-20] /apis/operators.coreos.com/v1/olmconfigs?allowWatchBookmarks=true&resourceVersion=22304&timeoutSeconds=385&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
21x [6m2.126366333s] [200-20] /apis/console.openshift.io/v1/consoleplugins?allowWatchBookmarks=true&resourceVersion=22230&timeout=9m58s&timeoutSeconds=598&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
Raw
console access before fix
deads@fedora:~/workspaces/cluster-debug-tools/src/github.com/openshift/cluster-debug-tools$ ./kubectl-dev_tool audit -f '/home/deads/Downloads/audit-logs(5)/9e67a938c529c2e045643d88bb48f105d0d45369e4e535d8e3b1163a47e22dca/audit_logs/kube-apiserver' -otop --by=verb --user=system:serviceaccount:openshift-console-operator:console-operator
had 11864 line read failures
count: 42740, first: 2024-01-23T19:57:32-05:00, last: 2024-01-23T22:09:49-05:00, duration: 2h12m16.819359s
Top 10 "CREATE" (of 41 total hits):
23x [1.320286304s] [409-22] /apis/console.openshift.io/v1/consolenotifications [system:serviceaccount:openshift-console-operator:console-operator]
5x [ 5.2882ms] [201-4] /api/v1/namespaces/openshift-console/configmaps [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 4.081ms] [201-1] /apis/console.openshift.io/v1/consoleclidownloads [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 12.119ms] [201-1] /api/v1/namespaces/openshift-console/services [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 15.534ms] [201-1] /apis/route.openshift.io/v1/namespaces/openshift-console/routes [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 34.0695ms] [201-1] /apis/apps/v1/namespaces/openshift-console/deployments [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 4.7715ms] [201-1] /apis/policy/v1/namespaces/openshift-console/poddisruptionbudgets [system:serviceaccount:openshift-console-operator:console-operator]
1x [ 16.05ms] [] /api/v1/namespaces/openshift-config-managed/configmaps [system:serviceaccount:openshift-console-operator:console-operator]
1x [ 3.283ms] [] /api/v1/namespaces/openshift-console/secrets [system:serviceaccount:openshift-console-operator:console-operator]
1x [ 3.032ms] [] /apis/coordination.k8s.io/v1/namespaces/openshift-console-operator/leases?timeout=1m47s [system:serviceaccount:openshift-console-operator:console-operator]
Top 10 "DELETE" (of 1390 total hits):
420x [ 6.40598ms] [404-419] /apis/route.openshift.io/v1/namespaces/openshift-console/routes/downloads-custom [system:serviceaccount:openshift-console-operator:console-operator]
420x [ 6.413904ms] [404-419] /apis/route.openshift.io/v1/namespaces/openshift-console/routes/console-custom [system:serviceaccount:openshift-console-operator:console-operator]
416x [ 85.320437ms] [404-414,504-1] /api/v1/namespaces/openshift-console/services/console-redirect [system:serviceaccount:openshift-console-operator:console-operator]
134x [ 3.067664ms] [404-133] /apis/console.openshift.io/v1/consolenotifications/cluster-upgrade [system:serviceaccount:openshift-console-operator:console-operator]
Top 10 "GET" (of 34009 total hits):
9826x [ 2.996636ms] [200-9824,404-1] /apis/apps/v1/namespaces/openshift-console/deployments/console [system:serviceaccount:openshift-console-operator:console-operator]
4913x [ 3.97492ms] [200-4912] /apis/operators.coreos.com/v1/olmconfigs/cluster [system:serviceaccount:openshift-console-operator:console-operator]
4913x [ 3.87838ms] [200-4912] /api/v1/namespaces/openshift-console/configmaps/console-config [system:serviceaccount:openshift-console-operator:console-operator]
4913x [ 2.744786ms] [200-4912] /api/v1/namespaces/openshift-config-managed/configmaps/console-public [system:serviceaccount:openshift-console-operator:console-operator]
2473x [ 63.935934ms] [200-2472] /apis/config.openshift.io/v1/ingresses/cluster [system:serviceaccount:openshift-console-operator:console-operator]
837x [ 5.322851ms] [200-835,404-1] /apis/route.openshift.io/v1/namespaces/openshift-console/routes/downloads [system:serviceaccount:openshift-console-operator:console-operator]
791x [ 5.132159ms] [200-790] /apis/route.openshift.io/v1/namespaces/openshift-console/routes/console [system:serviceaccount:openshift-console-operator:console-operator]
575x [ 32.76746ms] [200-555,404-19] /api/v1/namespaces/openshift-console/configmaps/trusted-ca-bundle [system:serviceaccount:openshift-console-operator:console-operator]
555x [110.718609ms] [200-553,500-1] /api/v1/namespaces/openshift-console/configmaps/default-ingress-cert [system:serviceaccount:openshift-console-operator:console-operator]
497x [ 16.630883ms] [200-496] /apis/oauth.openshift.io/v1/oauthclients/console [system:serviceaccount:openshift-console-operator:console-operator]
Top 10 "LIST" (of 6041 total hits):
4915x [ 19.173308ms] [200-4913,500-1] /api/v1/nodes?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
1090x [ 32.766629ms] [200-1089] /apis/console.openshift.io/v1/consoleplugins?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
3x [ 1.237666ms] [200-2] /apis/operators.coreos.com/v1/olmconfigs [system:serviceaccount:openshift-console-operator:console-operator]
3x [ 553.666µs] [200-2] /api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dextension-apiserver-authentication&limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 967µs] [200-1] /apis/operator.openshift.io/v1/consoles?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 459.5µs] [200-1] /apis/console.openshift.io/v1/consoleclidownloads?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 19.579ms] [200-1] /api/v1/namespaces/openshift-config-managed/configmaps?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 389µs] [200-1] /apis/config.openshift.io/v1/ingresses?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 1.81ms] [200-1] /api/v1/namespaces/openshift-console/configmaps?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 502µs] [200-1] /apis/config.openshift.io/v1/oauths?limit=500&resourceVersion=0 [system:serviceaccount:openshift-console-operator:console-operator]
Top 10 "UPDATE" (of 681 total hits):
345x [141.606272ms] [200-271,409-73] /apis/operator.openshift.io/v1/consoles/cluster/status [system:serviceaccount:openshift-console-operator:console-operator]
304x [ 3.917154ms] [200-303] /apis/coordination.k8s.io/v1/namespaces/openshift-console-operator/leases/console-operator-lock?timeout=1m47s [system:serviceaccount:openshift-console-operator:console-operator]
20x [ 195.59785ms] [200-17,409-2] /apis/config.openshift.io/v1/clusteroperators/console/status [system:serviceaccount:openshift-console-operator:console-operator]
6x [ 6.227833ms] [200-5] /apis/apps/v1/namespaces/openshift-console/deployments/console [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 3.852ms] [200-1] /api/v1/namespaces/openshift-console/configmaps/console-config [system:serviceaccount:openshift-console-operator:console-operator]
2x [ 4.6335ms] [200-1] /apis/apps/v1/namespaces/openshift-console/deployments/downloads [system:serviceaccount:openshift-console-operator:console-operator]
1x [ 20.27ms] [] /apis/oauth.openshift.io/v1/oauthclients/console [system:serviceaccount:openshift-console-operator:console-operator]
1x [ 25.288ms] [] /apis/config.openshift.io/v1/consoles/cluster/status [system:serviceaccount:openshift-console-operator:console-operator]
Top 10 "WATCH" (of 578 total hits):
56x [6m51.367672339s] [200-55] /api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dextension-apiserver-authentication&resourceVersion=21365&timeout=6m19s&timeoutSeconds=379&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
40x [6m25.4397443s] [200-39] /api/v1/namespaces/openshift-config-managed/configmaps?allowWatchBookmarks=true&resourceVersion=21365&timeout=6m30s&timeoutSeconds=390&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
39x [6m32.197175179s] [200-38] /api/v1/namespaces/openshift-console/configmaps?allowWatchBookmarks=true&resourceVersion=21365&timeout=9m56s&timeoutSeconds=596&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
38x [6m48.279472157s] [200-37] /api/v1/namespaces/openshift-config/configmaps?allowWatchBookmarks=true&resourceVersion=21365&timeout=9m9s&timeoutSeconds=549&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
21x [6m0.904159809s] [200-20] /apis/config.openshift.io/v1/featuregates?allowWatchBookmarks=true&resourceVersion=21264&timeout=9m14s&timeoutSeconds=554&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
21x [6m9.192188095s] [200-20] /apis/config.openshift.io/v1/consoles?allowWatchBookmarks=true&resourceVersion=21365&timeout=8m42s&timeoutSeconds=522&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
21x [6m1.525230523s] [200-20] /apis/config.openshift.io/v1/ingresses?allowWatchBookmarks=true&resourceVersion=21330&timeout=6m0s&timeoutSeconds=360&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
21x [6m14.147130523s] [200-20] /apis/console.openshift.io/v1/consoleclidownloads?allowWatchBookmarks=true&resourceVersion=21068&timeout=9m22s&timeoutSeconds=562&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
21x [6m6.095325809s] [200-20] /apis/console.openshift.io/v1/consoleplugins?allowWatchBookmarks=true&resourceVersion=21374&timeout=9m45s&timeoutSeconds=585&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
20x [6m26.8523001s] [200-19] /apis/config.openshift.io/v1/oauths?allowWatchBookmarks=true&resourceVersion=21271&timeout=9m25s&timeoutSeconds=565&watch=true [system:serviceaccount:openshift-console-operator:console-operator]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment