Skip to content

Instantly share code, notes, and snippets.

@praveenkumar

praveenkumar/[Dont DELETE] Openshift_Operators

Created October 17, 2019 10:08
Show Gist options
  • Save praveenkumar/b5ceab217b57e2fb04606dcb604e0017 to your computer and use it in GitHub Desktop.
Save praveenkumar/b5ceab217b57e2fb04606dcb604e0017 to your computer and use it in GitHub Desktop.
Download ZIP
Put the details about consumed resources by Openshift Operators.
Raw
[Dont DELETE] Openshift_Operators
This gist thread will be used to find out what all operators are present as part of CRC and what can be done to make low
memory/cpu footprint to the host.
@praveenkumar
Copy link
Author

praveenkumar commented Dec 20, 2019 •
edited
Loading

Disable all the operators and remove memory from the resources from all the deployments.

oc scale --replicas=0 deployment --all -n openshift-dns-operator
oc scale --replicas=0 deployment --all -n openshift-authentication-operator
oc scale --replicas=0 deployment --all -n openshift-cluster-node-tuning-operator
oc scale --replicas=0 deployment --all -n openshift-console-operator
oc scale --replicas=0 deployment cluster-image-registry-operator -n openshift-image-registry
oc scale --replicas=0 deployment --all -n openshift-ingress-operator
oc scale --replicas=0 deployment marketplace-operator -n openshift-marketplace
oc scale --replicas=0 deployment --all -n openshift-controller-manager-operator
oc scale --replicas=0 deployment --all -n openshift-apiserver-operator
oc scale --replicas=0 deployment --all -n openshift-kube-apiserver-operator
oc scale --replicas=0 deployment --all -n openshift-kube-controller-manager-operator
oc scale --replicas=0 deployment --all -n openshift-kube-scheduler-operator
oc scale --replicas=0 deployment --all -n openshift-machine-api
oc scale --replicas=0 deployment --all -n openshift-machine-config-operator
oc scale --replicas=0 deployment --all -n openshift-monitoring
oc scale --replicas=0 deployment --all -n openshift-service-ca-operator
oc scale --replicas=0 deployment --all -n openshift-service-catalog-apiserver-operator
oc patch deployment cloud-credential-operator -n openshift-cloud-credential-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-approver -n openshift-cluster-machine-approver --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-node-tuning-operator -n openshift-cluster-node-tuning-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-storage-operator -n openshift-cluster-storage-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-version-operator -n openshift-cluster-version --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment console-operator -n openshift-console-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment console -n openshift-console --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment downloads -n openshift-console --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment openshift-controller-manager-operator -n openshift-controller-manager-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment image-registry -n openshift-image-registry --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment router-default -n openshift-ingress --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment insights-operator -n openshift-insights --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment kube-apiserver-operator -n openshift-kube-apiserver-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment kube-controller-manager-operator -n openshift-kube-controller-manager-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment openshift-kube-scheduler-operator -n openshift-kube-scheduler-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-autoscaler-operator -n openshift-machine-api --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-api-controllers -n openshift-machine-api --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-api-operator -n openshift-machine-api --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment etcd-quorum-guard -n openshift-machine-config-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-config-controller -n openshift-machine-config-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment machine-config-operator -n openshift-machine-config-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment certified-operators -n openshift-marketplace --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment community-operators -n openshift-marketplace --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment marketplace-operator -n openshift-marketplace --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment redhat-operators -n openshift-marketplace --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment cluster-monitoring-operator -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment grafana -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment kube-state-metrics -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment openshift-state-metrics -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment prometheus-adapter -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment prometheus-operator -n openshift-monitoring --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment network-operator -n openshift-network-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment catalog-operator -n openshift-operator-lifecycle-manager --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment olm-operator -n openshift-operator-lifecycle-manager --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment packageserver -n openshift-operator-lifecycle-manager --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment service-ca-operator -n openshift-service-ca-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment apiservice-cabundle-injector -n openshift-service-ca --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment configmap-cabundle-injector -n openshift-service-ca --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment service-serving-cert-signer -n openshift-service-ca --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'
oc patch deployment openshift-service-catalog-controller-manager-operator -n openshift-service-catalog-controller-manager-operator --type=json -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/resources/requests/memory"}]'

@praveenkumar
Copy link
Author

Check all the pods in all the namespaces.

$ oc get pods -A
NAMESPACE                                               NAME                                                              READY   STATUS    RESTARTS   AGE
openshift-apiserver                                     apiserver-6spqp                                                   1/1     Running   0          7d21h
openshift-authentication                                oauth-openshift-5f47cd5fbd-7xrfg                                  1/1     Running   0          12h
openshift-authentication                                oauth-openshift-5f47cd5fbd-cz6jm                                  1/1     Running   0          12h
openshift-cluster-machine-approver                      machine-approver-dc7c6f989-gr2fc                                  1/1     Running   0          11h
openshift-cluster-node-tuning-operator                  tuned-v8fzh                                                       1/1     Running   0          17h
openshift-cluster-samples-operator                      cluster-samples-operator-5cff448f4c-8c7l8                         1/1     Running   0          7d22h
openshift-cluster-storage-operator                      cluster-storage-operator-68749dbccb-v7wvt                         1/1     Running   0          11h
openshift-console                                       console-59b777fbdd-5lmg8                                          1/1     Running   0          11h
openshift-controller-manager                            controller-manager-5c48l                                          1/1     Running   2          17h
openshift-dns                                           dns-default-s4vmh                                                 2/2     Running   0          7d23h
openshift-etcd                                          etcd-member-crc-rk2fc-master-0                                    2/2     Running   0          7d23h
openshift-image-registry                                image-registry-6d8db9f49-kl6z8                                    1/1     Running   0          11h
openshift-image-registry                                node-ca-wb4hl                                                     1/1     Running   0          7d22h
openshift-ingress                                       router-default-799b6c477d-hnjc4                                   1/1     Running   0          11h
openshift-kube-apiserver                                kube-apiserver-crc-rk2fc-master-0                                 3/3     Running   11         17h
openshift-kube-controller-manager                       kube-controller-manager-crc-rk2fc-master-0                        2/2     Running   11         7d22h
openshift-kube-scheduler                                openshift-kube-scheduler-crc-rk2fc-master-0                       1/1     Running   11         7d22h
openshift-machine-config-operator                       machine-config-daemon-z7hlr                                       1/1     Running   0          7d23h
openshift-machine-config-operator                       machine-config-server-gq9ww                                       1/1     Running   0          7d23h
openshift-marketplace                                   certified-operators-6c45d48df5-2qn75                              1/1     Running   0          11h
openshift-marketplace                                   community-operators-7ff94d5784-h9xwb                              1/1     Running   0          11h
openshift-marketplace                                   redhat-operators-5c559c7bc-8tvvc                                  1/1     Running   0          11h
openshift-monitoring                                    node-exporter-cnt7b                                               2/2     Running   0          7d22h
openshift-multus                                        multus-admission-controller-68qmm                                 1/1     Running   0          7d23h
openshift-multus                                        multus-qjs27                                                      1/1     Running   3          7d23h
openshift-network-operator                              network-operator-64f87884fb-mgqbf                                 1/1     Running   0          11h
openshift-operator-lifecycle-manager                    catalog-operator-84f749ffc-ljttf                                  1/1     Running   0          11h
openshift-operator-lifecycle-manager                    olm-operator-7fc87c9c64-fm4c7                                     1/1     Running   0          11h
openshift-operator-lifecycle-manager                    packageserver-848764c995-674n7                                    1/1     Running   0          4m14s
openshift-operator-lifecycle-manager                    packageserver-848764c995-zs57p                                    1/1     Running   0          4m
openshift-sdn                                           ovs-42ltn                                                         1/1     Running   0          7d23h
openshift-sdn                                           sdn-controller-wp95c                                              1/1     Running   1          7d23h
openshift-sdn                                           sdn-jf826                                                         1/1     Running   12         7d23h
openshift-service-ca                                    apiservice-cabundle-injector-6dd5968b98-fgmmw                     1/1     Running   0          4m3s
openshift-service-ca                                    configmap-cabundle-injector-5bc9fb4464-bm7ch                      1/1     Running   0          4m5s
openshift-service-ca                                    service-serving-cert-signer-55f844f768-d5wgz                      1/1     Running   0          4m5s
openshift-service-catalog-controller-manager-operator   openshift-service-catalog-controller-manager-operator-68765fwqd   1/1     Running   0          11h

@praveenkumar
Copy link
Author

Check crictl stats now, which is much better.

# crictl stats
CONTAINER           CPU %               MEM                 DISK                INODES
198d7391b4187       0.08                23.19MB             0B                  0
203e7292e268e       1.06                151.1MB             0B                  0
20b0c496a153b       0.14                49.89MB             0B                  0
230126ec6bd44       0.05                17.52MB             0B                  0
26019a315efd5       0.04                26.34MB             0B                  0
2afc10a0a9ffc       0.13                35.59MB             0B                  0
2bb066aa6cb75       0.05                24.15MB             0B                  0
2cf3b7299ea4b       0.19                70.73MB             0B                  0
318b795029536       0.14                65.71MB             0B                  0
3467759e17ea5       0.00                44.22MB             0B                  0
39149ade182fe       0.00                28.85MB             0B                  0
5e0d102aa9e42       1.09                77.8MB              0B                  0
5eb215537672b       0.00                32.52MB             0B                  0
610dc0e2d6cbc       3.86                19.36MB             0B                  0
6aaf04f44a1a5       0.15                34.22MB             0B                  0
7d157acf3f9d2       5.96                417.7MB             0B                  0
7d73fd9db859b       0.07                77.99MB             0B                  0
7d917841c69b7       0.00                8.802MB             0B                  0
915e30ac44a4f       0.02                58.49MB             0B                  0
9338c3612c43d       0.08                21.99MB             0B                  0
982ac5fbcfe80       0.68                38.44MB             0B                  0
9c0b8138ab691       0.00                22.35MB             0B                  0
9cd7a685ef01b       0.00                8.684MB             0B                  0
9ed0eb3cf5c86       0.00                3.461MB             0B                  0
a1bb5331bcd55       2.07                197.4MB             0B                  0
a89d53e321f46       1.03                76.6MB              0B                  0
a8df1963598e1       1.76                291.4MB             0B                  0
ac25188f73d73       0.01                34.94MB             0B                  0
af73819bd4f02       0.72                123.2MB             0B                  0
b1d82219d1052       0.25                55.63MB             0B                  0
b2c2b2e67729e       0.46                79.63MB             0B                  0
b75b2c3cbc87c       0.00                8.978MB             0B                  0
bb1b6db3b5545       0.12                92.76MB             0B                  0
cb45728dfe7b4       0.14                11.26MB             0B                  0
cbe6b8d825427       0.27                53.76MB             0B                  0
d0e583cf88b7e       1.21                69.3MB              0B                  0
d478d4e39655d       0.07                41.65MB             0B                  0
d62367c891017       0.00                46.65MB             0B                  0
d66a577660254       0.02                233.9MB             0B                  0
dd087ff88beb3       5.75                1.917GB             0B                  0
df13e81d945f3       0.07                31.79MB             0B                  0
e45b2c55a9391       0.03                12.25MB             0B                  0
e6b3eee5fd7ba       0.00                31.13MB             0B                  0
f375fe4ffbe94       0.20                24.46MB             0B                  0

@praveenkumar
Copy link
Author

praveenkumar commented Dec 20, 2019 •
edited
Loading

Now try to deploy the knative serving and eventing operator on this cluster.

$ cat sub.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: knative-eventing-operator
  namespace: openshift-operators 
spec:
  channel: alpha
  name: knative-eventing-operator
  source: community-operators
  sourceNamespace: openshift-marketplace 

$ cat sub-serving.yaml 
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: knative-serving-operator
  namespace: openshift-operators 
spec:
  channel: alpha
  name: knative-serving-operator
  source: community-operators
  sourceNamespace: openshift-marketplace 

$ oc apply -f sub.yaml
$ oc apply -f sub-serving.yaml

Wait for some time and check the pods (which all are running).

$ oc get pods -A
NAMESPACE                                               NAME                                                              READY   STATUS    RESTARTS   AGE
istio-operator                                          istio-operator-86f7d5ffd8-ssmwq                                   1/1     Running   0          108m
istio-system                                            cluster-local-gateway-58758f588-wzkm2                             1/1     Running   0          106m
istio-system                                            istio-ingressgateway-54968c8854-fx97t                             1/1     Running   0          106m
istio-system                                            istio-pilot-577ff6784c-jrjjb                                      1/1     Running   0          107m
knative-eventing                                        eventing-controller-5d5f979874-96kpb                              1/1     Running   0          114m
knative-eventing                                        eventing-webhook-75bcb6d4bb-j5tml                                 1/1     Running   0          114m
knative-eventing                                        imc-controller-69c54bfdc8-pzxlr                                   1/1     Running   0          113m
knative-eventing                                        imc-dispatcher-94bc9f6b6-c2kfl                                    1/1     Running   0          113m
knative-eventing                                        sources-controller-5c6df78ffb-g9qhj                               1/1     Running   0          114m
knative-serving                                         activator-6c94b9ff47-9xwlp                                        1/1     Running   0          106m
knative-serving                                         autoscaler-64c549bcf4-lhfrx                                       1/1     Running   0          106m
knative-serving                                         controller-564487997f-mh2l2                                       1/1     Running   0          106m
knative-serving                                         knative-openshift-ingress-c7fd864df-7k6xw                         1/1     Running   0          106m
knative-serving                                         networking-certmanager-8466b656dc-ktf9d                           1/1     Running   0          106m
knative-serving                                         networking-istio-59f5c789d-9j69g                                  1/1     Running   0          106m
knative-serving                                         webhook-77ff9bd584-wwz4j                                          1/1     Running   0          106m
[...]

Try to deploy a sample service and check.

$ cat service.yaml 
apiVersion: serving.knative.dev/v1alpha1
kind: Service
metadata:
  name: helloworld-go
  namespace: default
spec:
  template:
    spec:
      containers:
        - image: gcr.io/knative-samples/helloworld-go
          env:
            - name: TARGET
              value: "Go Sample v1"

$ oc get ksvc helloworld-go
NAME            URL                                             LATESTCREATED         LATESTREADY           READY   REASON
helloworld-go   http://helloworld-go.default.apps-crc.testing   helloworld-go-lk6g9   helloworld-go-lk6g9   True    

$ curl http://helloworld-go.default.apps-crc.testing
Hello Go Sample v1!

$ oc describe node
[...]
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests          Limits
  --------           --------          ------
  cpu                3135m (89%)       4700m (134%)
  memory             6435496192 (87%)  5140890112 (70%)
  ephemeral-storage  0 (0%)            0 (0%)
Events:
  Type    Reason                   Age                 From                         Message
  ----    ------                   ----                ----                         -------
  Normal  NodeHasSufficientMemory  49m (x37 over 20h)  kubelet, crc-rk2fc-master-0  Node crc-rk2fc-master-0 status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    49m (x37 over 20h)  kubelet, crc-rk2fc-master-0  Node crc-rk2fc-master-0 status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     49m (x37 over 20h)  kubelet, crc-rk2fc-master-0  Node crc-rk2fc-master-0 status is now: NodeHasSufficientPID
  Normal  NodeReady                49m (x5 over 20h)   kubelet, crc-rk2fc-master-0  Node crc-rk2fc-master-0 status is now: NodeReady

Check the load on the CRC VM.

sh-4.2# chroot /host 
sh-4.4# free -h
              total        used        free      shared  buff/cache   available
Mem:          7.4Gi       4.3Gi       885Mi       248Mi       2.3Gi       3.4Gi
Swap:            0B          0B          0B

sh-4.4# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3        30G   19G   12G  62% /
tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
devtmpfs        3.7G     0  3.7G   0% /dev
tmpfs           3.8G   84K  3.8G   1% /dev/shm
tmpfs           3.8G  8.6M  3.8G   1% /run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment