Created
October 17, 2019 10:08
-
-
Save praveenkumar/b5ceab217b57e2fb04606dcb604e0017 to your computer and use it in GitHub Desktop.
Put the details about consumed resources by Openshift Operators.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| This gist thread will be used to find out what all operators are present as part of CRC and what can be done to make low | |
| memory/cpu footprint to the host. |
Author
praveenkumar
commented
Dec 19, 2019
For minikube 1.17.0 k8s version
$ sudo crictl stats
CONTAINER CPU % MEM DISK INODES
06d0e4252080d 0.20 46.19MB 0B 0
1c1964ee0bfaa 0.05 65.5MB 0B 0
26f1c832b0b97 1.63 57.72MB 0B 0
2b3ee349f435d 0.00 39.67MB 0B 0
52b8b416afc39 0.18 7.524MB 0B 0
65e73259f88cd 1.03 28.21MB 0B 0
9cbed8996876b 0.00 43.65MB 0B 0
c2326ddf927e9 2.39 271.3MB 0B 0
e9e92bed617c2 0.47 38.87MB 0B 0
$ sudo crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/kubernetesui/dashboard v2.0.0-beta8 eb51a35975256 91.2MB
docker.io/kubernetesui/metrics-scraper v1.0.2 3b08661dc379d 40.1MB
gcr.io/k8s-minikube/storage-provisioner v1.8.1 4689081edb103 80.8MB
k8s.gcr.io/coredns 1.6.5 70f311871ae12 41.7MB
k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90d 290MB
k8s.gcr.io/kube-addon-manager v9.0.2 bd12a212f9dcb 84.7MB
k8s.gcr.io/kube-apiserver v1.17.0 0cae8d5cc64c7 173MB
k8s.gcr.io/kube-controller-manager v1.17.0 5eb3b74868724 162MB
k8s.gcr.io/kube-proxy v1.17.0 7d54289267dc5 118MB
k8s.gcr.io/kube-scheduler v1.17.0 78c190f736b11 96MB
k8s.gcr.io/pause 3.1 da86e6ba6ca19 747kB
Disable all the operators and remove memory from the resources from all the deployments.
oc scale --replicas=0 deployment --all -n openshift-dns-operator
oc scale --replicas=0 deployment --all -n openshift-authentication-operator
oc scale --replicas=0 deployment --all -n openshift-cluster-node-tuning-operator
oc scale --replicas=0 deployment --all -n openshift-console-operator
oc scale --replicas=0 deployment cluster-image-registry-operator -n openshift-image-registry
oc scale --replicas=0 deployment --all -n openshift-ingress-operator
oc scale --replicas=0 deployment marketplace-operator -n openshift-marketplace
oc scale --replicas=0 deployment --all -n openshift-controller-manager-operator
oc scale --replicas=0 deployment --all -n openshift-apiserver-operator
oc scale --replicas=0 deployment --all -n openshift-kube-apiserver-operator
oc scale --replicas=0 deployment --all -n openshift-kube-controller-manager-operator
oc scale --replicas=0 deployment --all -n openshift-kube-scheduler-operator
oc scale --replicas=0 deployment --all -n openshift-machine-api
oc scale --replicas=0 deployment --all -n openshift-machine-config-operator
oc scale --replicas=0 deployment --all -n openshift-monitoring
oc scale --replicas=0 deployment --all -n openshift-service-ca-operator
oc scale --replicas=0 deployment --all -n openshift-service-catalog-apiserver-operator
oc patch deployment cloud-credential-operator -n openshift-cloud-credential-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-approver -n openshift-cluster-machine-approver --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-node-tuning-operator -n openshift-cluster-node-tuning-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-storage-operator -n openshift-cluster-storage-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-version-operator -n openshift-cluster-version --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment console-operator -n openshift-console-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment console -n openshift-console --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment downloads -n openshift-console --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment openshift-controller-manager-operator -n openshift-controller-manager-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment image-registry -n openshift-image-registry --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment router-default -n openshift-ingress --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment insights-operator -n openshift-insights --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment kube-apiserver-operator -n openshift-kube-apiserver-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment kube-controller-manager-operator -n openshift-kube-controller-manager-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment openshift-kube-scheduler-operator -n openshift-kube-scheduler-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-autoscaler-operator -n openshift-machine-api --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-api-controllers -n openshift-machine-api --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-api-operator -n openshift-machine-api --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment etcd-quorum-guard -n openshift-machine-config-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-config-controller -n openshift-machine-config-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-config-operator -n openshift-machine-config-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment certified-operators -n openshift-marketplace --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment community-operators -n openshift-marketplace --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment marketplace-operator -n openshift-marketplace --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment redhat-operators -n openshift-marketplace --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-monitoring-operator -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment grafana -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment kube-state-metrics -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment openshift-state-metrics -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment prometheus-adapter -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment prometheus-operator -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment network-operator -n openshift-network-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment catalog-operator -n openshift-operator-lifecycle-manager --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment olm-operator -n openshift-operator-lifecycle-manager --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment packageserver -n openshift-operator-lifecycle-manager --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment service-ca-operator -n openshift-service-ca-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment apiservice-cabundle-injector -n openshift-service-ca --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment configmap-cabundle-injector -n openshift-service-ca --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment service-serving-cert-signer -n openshift-service-ca --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment openshift-service-catalog-controller-manager-operator -n openshift-service-catalog-controller-manager-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
Check all the pods in all the namespaces.
$ oc get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
openshift-apiserver apiserver-6spqp 1/1 Running 0 7d21h
openshift-authentication oauth-openshift-5f47cd5fbd-7xrfg 1/1 Running 0 12h
openshift-authentication oauth-openshift-5f47cd5fbd-cz6jm 1/1 Running 0 12h
openshift-cluster-machine-approver machine-approver-dc7c6f989-gr2fc 1/1 Running 0 11h
openshift-cluster-node-tuning-operator tuned-v8fzh 1/1 Running 0 17h
openshift-cluster-samples-operator cluster-samples-operator-5cff448f4c-8c7l8 1/1 Running 0 7d22h
openshift-cluster-storage-operator cluster-storage-operator-68749dbccb-v7wvt 1/1 Running 0 11h
openshift-console console-59b777fbdd-5lmg8 1/1 Running 0 11h
openshift-controller-manager controller-manager-5c48l 1/1 Running 2 17h
openshift-dns dns-default-s4vmh 2/2 Running 0 7d23h
openshift-etcd etcd-member-crc-rk2fc-master-0 2/2 Running 0 7d23h
openshift-image-registry image-registry-6d8db9f49-kl6z8 1/1 Running 0 11h
openshift-image-registry node-ca-wb4hl 1/1 Running 0 7d22h
openshift-ingress router-default-799b6c477d-hnjc4 1/1 Running 0 11h
openshift-kube-apiserver kube-apiserver-crc-rk2fc-master-0 3/3 Running 11 17h
openshift-kube-controller-manager kube-controller-manager-crc-rk2fc-master-0 2/2 Running 11 7d22h
openshift-kube-scheduler openshift-kube-scheduler-crc-rk2fc-master-0 1/1 Running 11 7d22h
openshift-machine-config-operator machine-config-daemon-z7hlr 1/1 Running 0 7d23h
openshift-machine-config-operator machine-config-server-gq9ww 1/1 Running 0 7d23h
openshift-marketplace certified-operators-6c45d48df5-2qn75 1/1 Running 0 11h
openshift-marketplace community-operators-7ff94d5784-h9xwb 1/1 Running 0 11h
openshift-marketplace redhat-operators-5c559c7bc-8tvvc 1/1 Running 0 11h
openshift-monitoring node-exporter-cnt7b 2/2 Running 0 7d22h
openshift-multus multus-admission-controller-68qmm 1/1 Running 0 7d23h
openshift-multus multus-qjs27 1/1 Running 3 7d23h
openshift-network-operator network-operator-64f87884fb-mgqbf 1/1 Running 0 11h
openshift-operator-lifecycle-manager catalog-operator-84f749ffc-ljttf 1/1 Running 0 11h
openshift-operator-lifecycle-manager olm-operator-7fc87c9c64-fm4c7 1/1 Running 0 11h
openshift-operator-lifecycle-manager packageserver-848764c995-674n7 1/1 Running 0 4m14s
openshift-operator-lifecycle-manager packageserver-848764c995-zs57p 1/1 Running 0 4m
openshift-sdn ovs-42ltn 1/1 Running 0 7d23h
openshift-sdn sdn-controller-wp95c 1/1 Running 1 7d23h
openshift-sdn sdn-jf826 1/1 Running 12 7d23h
openshift-service-ca apiservice-cabundle-injector-6dd5968b98-fgmmw 1/1 Running 0 4m3s
openshift-service-ca configmap-cabundle-injector-5bc9fb4464-bm7ch 1/1 Running 0 4m5s
openshift-service-ca service-serving-cert-signer-55f844f768-d5wgz 1/1 Running 0 4m5s
openshift-service-catalog-controller-manager-operator openshift-service-catalog-controller-manager-operator-68765fwqd 1/1 Running 0 11h
Check crictl stats now, which is much better.
# crictl stats
CONTAINER CPU % MEM DISK INODES
198d7391b4187 0.08 23.19MB 0B 0
203e7292e268e 1.06 151.1MB 0B 0
20b0c496a153b 0.14 49.89MB 0B 0
230126ec6bd44 0.05 17.52MB 0B 0
26019a315efd5 0.04 26.34MB 0B 0
2afc10a0a9ffc 0.13 35.59MB 0B 0
2bb066aa6cb75 0.05 24.15MB 0B 0
2cf3b7299ea4b 0.19 70.73MB 0B 0
318b795029536 0.14 65.71MB 0B 0
3467759e17ea5 0.00 44.22MB 0B 0
39149ade182fe 0.00 28.85MB 0B 0
5e0d102aa9e42 1.09 77.8MB 0B 0
5eb215537672b 0.00 32.52MB 0B 0
610dc0e2d6cbc 3.86 19.36MB 0B 0
6aaf04f44a1a5 0.15 34.22MB 0B 0
7d157acf3f9d2 5.96 417.7MB 0B 0
7d73fd9db859b 0.07 77.99MB 0B 0
7d917841c69b7 0.00 8.802MB 0B 0
915e30ac44a4f 0.02 58.49MB 0B 0
9338c3612c43d 0.08 21.99MB 0B 0
982ac5fbcfe80 0.68 38.44MB 0B 0
9c0b8138ab691 0.00 22.35MB 0B 0
9cd7a685ef01b 0.00 8.684MB 0B 0
9ed0eb3cf5c86 0.00 3.461MB 0B 0
a1bb5331bcd55 2.07 197.4MB 0B 0
a89d53e321f46 1.03 76.6MB 0B 0
a8df1963598e1 1.76 291.4MB 0B 0
ac25188f73d73 0.01 34.94MB 0B 0
af73819bd4f02 0.72 123.2MB 0B 0
b1d82219d1052 0.25 55.63MB 0B 0
b2c2b2e67729e 0.46 79.63MB 0B 0
b75b2c3cbc87c 0.00 8.978MB 0B 0
bb1b6db3b5545 0.12 92.76MB 0B 0
cb45728dfe7b4 0.14 11.26MB 0B 0
cbe6b8d825427 0.27 53.76MB 0B 0
d0e583cf88b7e 1.21 69.3MB 0B 0
d478d4e39655d 0.07 41.65MB 0B 0
d62367c891017 0.00 46.65MB 0B 0
d66a577660254 0.02 233.9MB 0B 0
dd087ff88beb3 5.75 1.917GB 0B 0
df13e81d945f3 0.07 31.79MB 0B 0
e45b2c55a9391 0.03 12.25MB 0B 0
e6b3eee5fd7ba 0.00 31.13MB 0B 0
f375fe4ffbe94 0.20 24.46MB 0B 0
Now try to deploy the knative serving and eventing operator on this cluster.
$ cat sub.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: knative-eventing-operator
namespace: openshift-operators
spec:
channel: alpha
name: knative-eventing-operator
source: community-operators
sourceNamespace: openshift-marketplace
$ cat sub-serving.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: knative-serving-operator
namespace: openshift-operators
spec:
channel: alpha
name: knative-serving-operator
source: community-operators
sourceNamespace: openshift-marketplace
$ oc apply -f sub.yaml
$ oc apply -f sub-serving.yaml
Wait for some time and check the pods (which all are running).
$ oc get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
istio-operator istio-operator-86f7d5ffd8-ssmwq 1/1 Running 0 108m
istio-system cluster-local-gateway-58758f588-wzkm2 1/1 Running 0 106m
istio-system istio-ingressgateway-54968c8854-fx97t 1/1 Running 0 106m
istio-system istio-pilot-577ff6784c-jrjjb 1/1 Running 0 107m
knative-eventing eventing-controller-5d5f979874-96kpb 1/1 Running 0 114m
knative-eventing eventing-webhook-75bcb6d4bb-j5tml 1/1 Running 0 114m
knative-eventing imc-controller-69c54bfdc8-pzxlr 1/1 Running 0 113m
knative-eventing imc-dispatcher-94bc9f6b6-c2kfl 1/1 Running 0 113m
knative-eventing sources-controller-5c6df78ffb-g9qhj 1/1 Running 0 114m
knative-serving activator-6c94b9ff47-9xwlp 1/1 Running 0 106m
knative-serving autoscaler-64c549bcf4-lhfrx 1/1 Running 0 106m
knative-serving controller-564487997f-mh2l2 1/1 Running 0 106m
knative-serving knative-openshift-ingress-c7fd864df-7k6xw 1/1 Running 0 106m
knative-serving networking-certmanager-8466b656dc-ktf9d 1/1 Running 0 106m
knative-serving networking-istio-59f5c789d-9j69g 1/1 Running 0 106m
knative-serving webhook-77ff9bd584-wwz4j 1/1 Running 0 106m
[...]
Try to deploy a sample service and check.
$ cat service.yaml
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
name: helloworld-go
namespace: default
spec:
template:
spec:
containers:
- image: gcr.io/knative-samples/helloworld-go
env:
- name: TARGET
value: "Go Sample v1"
$ oc get ksvc helloworld-go
NAME URL LATESTCREATED LATESTREADY READY REASON
helloworld-go http://helloworld-go.default.apps-crc.testing helloworld-go-lk6g9 helloworld-go-lk6g9 True
$ curl http://helloworld-go.default.apps-crc.testing
Hello Go Sample v1!
$ oc describe node
[...]
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 3135m (89%) 4700m (134%)
memory 6435496192 (87%) 5140890112 (70%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasSufficientMemory 49m (x37 over 20h) kubelet, crc-rk2fc-master-0 Node crc-rk2fc-master-0 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 49m (x37 over 20h) kubelet, crc-rk2fc-master-0 Node crc-rk2fc-master-0 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 49m (x37 over 20h) kubelet, crc-rk2fc-master-0 Node crc-rk2fc-master-0 status is now: NodeHasSufficientPID
Normal NodeReady 49m (x5 over 20h) kubelet, crc-rk2fc-master-0 Node crc-rk2fc-master-0 status is now: NodeReady
Check the load on the CRC VM.
sh-4.2# chroot /host
sh-4.4# free -h
total used free shared buff/cache available
Mem: 7.4Gi 4.3Gi 885Mi 248Mi 2.3Gi 3.4Gi
Swap: 0B 0B 0B
sh-4.4# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda3 30G 19G 12G 62% /
tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup
devtmpfs 3.7G 0 3.7G 0% /dev
tmpfs 3.8G 84K 3.8G 1% /dev/shm
tmpfs 3.8G 8.6M 3.8G 1% /run
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment