Skip to content

Instantly share code, notes, and snippets.

@praveenkumar
Created November 9, 2018 08:46
Show Gist options
  • Save praveenkumar/3257f9d0e54d1d53b7aabe6da52a3671 to your computer and use it in GitHub Desktop.
Save praveenkumar/3257f9d0e54d1d53b7aabe6da52a3671 to your computer and use it in GitHub Desktop.
Installer master node start => stop => start
[core@test1-master-0 ~]$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-member-test1-master-0 1/1 Running 0 1h
kube-system kube-flannel-8jq4j 2/2 Running 2 1h
kube-system kube-flannel-qmn2c 2/2 Running 0 1h
kube-system kube-proxy-jdzzz 1/1 Running 0 1h
kube-system kube-proxy-mfstb 1/1 Running 0 1h
kube-system kube-scheduler-qgbm4 1/1 Running 0 1h
kube-system metrics-server-5767bfc576-nhg6w 2/2 Running 0 1h
kube-system tectonic-network-operator-cwqrd 1/1 Running 1 1h
openshift-apiserver apiserver-8l9tv 1/1 Running 0 15m
openshift-cluster-api clusterapi-manager-controllers-684c97f56b-h4dk2 3/3 Running 0 12m
openshift-cluster-api machine-api-operator-c8947696b-lpd74 1/1 Running 0 14m
openshift-cluster-dns-operator cluster-dns-operator-dbb6665b5-4gxf2 1/1 Running 0 24m
openshift-cluster-dns dns-default-5wgxm 1/1 Running 0 1h
openshift-cluster-dns dns-default-wxhrb 1/1 Running 0 1h
openshift-cluster-network-operator cluster-network-operator-k2qb9 1/1 Running 0 24m
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-5f7c598d5c-4wl8s 1/1 Running 0 13m
openshift-cluster-node-tuning tuned-5g6rv 1/1 Running 0 12m
openshift-cluster-node-tuning tuned-7hm86 1/1 Running 0 11m
openshift-cluster-samples-operator cluster-samples-operator-76d79b8c6c-jsxvd 1/1 Running 0 12m
openshift-cluster-version cluster-version-operator-824mk 1/1 Running 5 1h
openshift-console console-operator-7cb96f95cc-7hgt7 1/1 Running 0 11m
openshift-console openshift-console-669f887744-sdjwm 1/1 Running 0 54m
openshift-controller-manager controller-manager-qgws4 1/1 Running 0 15m
openshift-core-operators openshift-cluster-kube-apiserver-operator-75494fbfb8-pnzhs 1/1 Running 4 23m
openshift-core-operators openshift-cluster-kube-controller-manager-operator-597fcdfrw4rz 1/1 Running 4 22m
openshift-core-operators openshift-cluster-kube-scheduler-operator-657cffdbdd-5rh4b 1/1 Running 4 22m
openshift-core-operators openshift-cluster-openshift-apiserver-operator-f7546c66d-c57pl 1/1 Running 0 16m
openshift-core-operators openshift-cluster-openshift-controller-manager-operator-5fwgq25 1/1 Running 0 16m
openshift-core-operators openshift-service-cert-signer-operator-776d49f685-pwqcm 1/1 Running 1 23m
openshift-csi-operator csi-operator-5df7749cd9-s57kl 1/1 Running 0 10m
openshift-image-registry cluster-image-registry-operator-f95c59b44-b469t 1/1 Running 0 14m
openshift-ingress-operator ingress-operator-778475444b-5z27x 1/1 Running 0 13m
openshift-ingress router-default-7q9ct 1/1 Running 0 57m
openshift-kube-apiserver installer-1-test1-master-0 0/1 Completed 0 1h
openshift-kube-apiserver installer-2-test1-master-0 0/1 Completed 0 22m
openshift-kube-apiserver openshift-kube-apiserver-test1-master-0 1/1 Running 0 19m
openshift-kube-controller-manager installer-1-test1-master-0 0/1 Completed 0 1h
openshift-kube-controller-manager installer-2-test1-master-0 0/1 Completed 0 19m
openshift-kube-controller-manager openshift-kube-controller-manager-test1-master-0 1/1 Running 0 18m
openshift-kube-scheduler installer-1-test1-master-0 0/1 Completed 0 1h
openshift-kube-scheduler installer-2-test1-master-0 0/1 Completed 0 18m
openshift-kube-scheduler openshift-kube-scheduler-test1-master-0 1/1 Running 0 18m
openshift-machine-config-operator machine-config-controller-6cdb889b98-s7kp2 1/1 Running 0 14m
openshift-machine-config-operator machine-config-daemon-h6x8k 1/1 Running 0 11m
openshift-machine-config-operator machine-config-daemon-tksk7 1/1 Running 0 11m
openshift-machine-config-operator machine-config-operator-6c654dd6bd-fvsmz 1/1 Running 0 14m
openshift-machine-config-operator machine-config-server-q8rw8 1/1 Running 0 12m
openshift-monitoring cluster-monitoring-operator-7744c5fd65-t9dc2 1/1 Running 0 13m
openshift-monitoring grafana-57f595895d-rmwz9 2/2 Running 0 50m
openshift-monitoring prometheus-k8s-0 6/6 Running 2 46m
openshift-monitoring prometheus-k8s-1 6/6 Running 1 43m
openshift-monitoring prometheus-operator-99d89bcd-q2bfl 1/1 Running 0 47m
openshift-operator-lifecycle-manager catalog-operator-9758dfdc5-sf2mj 1/1 Running 0 15m
openshift-operator-lifecycle-manager olm-operator-67f6b68849-48trm 1/1 Running 0 15m
openshift-operator-lifecycle-manager package-server-79fb56977f-wh4gz 1/1 Running 0 14m
openshift-service-cert-signer apiservice-cabundle-injector-ccc4949d4-kxmws 1/1 Running 1 22m
openshift-service-cert-signer configmap-cabundle-injector-7598544959-cf9z9 1/1 Running 1 22m
openshift-service-cert-signer service-serving-cert-signer-785f84ccb9-jx2lk 1/1 Running 1 22m
tectonic-system kube-addon-operator-654588755f-hz65v 1/1 Running 1 1h
[core@test1-master-0 ~]$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
test1-master-0 Ready master 1h v1.11.0+d4cacc0
test1-worker-0-zd7hd NotReady worker 1h v1.11.0+d4cacc0
[core@test1-master-0 ~]$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-member-test1-master-0 1/1 Running 0 2h
kube-system kube-flannel-8jq4j 2/2 NodeLost 2 1h
kube-system kube-flannel-qmn2c 2/2 Running 0 1h
kube-system kube-proxy-jdzzz 1/1 Running 0 2h
kube-system kube-proxy-mfstb 1/1 NodeLost 0 1h
kube-system kube-scheduler-qgbm4 1/1 Running 0 2h
kube-system metrics-server-5767bfc576-lfrkd 0/2 Pending 0 3m
kube-system metrics-server-5767bfc576-nhg6w 2/2 Unknown 0 1h
kube-system tectonic-network-operator-cwqrd 1/1 Running 1 2h
openshift-apiserver apiserver-8l9tv 1/1 Running 0 21m
openshift-cluster-api clusterapi-manager-controllers-684c97f56b-h4dk2 3/3 Running 0 17m
openshift-cluster-api machine-api-operator-c8947696b-lpd74 1/1 Running 0 20m
openshift-cluster-dns-operator cluster-dns-operator-dbb6665b5-4gxf2 1/1 Unknown 0 29m
openshift-cluster-dns-operator cluster-dns-operator-dbb6665b5-v4wrp 1/1 Running 0 3m
openshift-cluster-dns dns-default-5wgxm 1/1 NodeLost 0 1h
openshift-cluster-dns dns-default-wxhrb 1/1 Running 0 1h
openshift-cluster-network-operator cluster-network-operator-k2qb9 1/1 Running 0 30m
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-5f7c598d5c-4wl8s 1/1 Unknown 0 18m
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-5f7c598d5c-jwjst 0/1 Pending 0 3m
openshift-cluster-node-tuning tuned-5g6rv 1/1 NodeLost 0 17m
openshift-cluster-node-tuning tuned-7hm86 1/1 Running 0 16m
openshift-cluster-samples-operator cluster-samples-operator-76d79b8c6c-gmkqd 1/1 Running 0 3m
openshift-cluster-samples-operator cluster-samples-operator-76d79b8c6c-jsxvd 1/1 Unknown 0 17m
openshift-cluster-version cluster-version-operator-824mk 1/1 Running 5 2h
openshift-console console-operator-7cb96f95cc-7hgt7 1/1 Unknown 0 16m
openshift-console console-operator-7cb96f95cc-zt6p5 0/1 Pending 0 3m
openshift-console openshift-console-669f887744-2qrzz 0/1 Pending 0 3m
openshift-console openshift-console-669f887744-sdjwm 1/1 Unknown 0 1h
openshift-controller-manager controller-manager-qgws4 1/1 Running 0 20m
openshift-core-operators openshift-cluster-kube-apiserver-operator-75494fbfb8-pnzhs 1/1 Running 4 28m
openshift-core-operators openshift-cluster-kube-controller-manager-operator-597fcdfrw4rz 1/1 Running 4 27m
openshift-core-operators openshift-cluster-kube-scheduler-operator-657cffdbdd-5rh4b 1/1 Running 4 27m
openshift-core-operators openshift-cluster-openshift-apiserver-operator-f7546c66d-c57pl 1/1 Running 0 22m
openshift-core-operators openshift-cluster-openshift-controller-manager-operator-5fwgq25 1/1 Running 0 21m
openshift-core-operators openshift-service-cert-signer-operator-776d49f685-pwqcm 1/1 Running 1 29m
openshift-csi-operator csi-operator-5df7749cd9-s57kl 1/1 Unknown 0 16m
openshift-csi-operator csi-operator-5df7749cd9-vnww4 0/1 Pending 0 3m
openshift-image-registry cluster-image-registry-operator-f95c59b44-b469t 1/1 Unknown 0 19m
openshift-image-registry cluster-image-registry-operator-f95c59b44-d6gq5 1/1 Running 0 3m
openshift-ingress-operator ingress-operator-778475444b-5z27x 1/1 Unknown 0 18m
openshift-ingress-operator ingress-operator-778475444b-789x4 0/1 Pending 0 3m
openshift-ingress router-default-7q9ct 1/1 NodeLost 0 1h
openshift-kube-apiserver installer-1-test1-master-0 0/1 Completed 0 1h
openshift-kube-apiserver installer-2-test1-master-0 0/1 Completed 0 27m
openshift-kube-apiserver openshift-kube-apiserver-test1-master-0 1/1 Running 0 25m
openshift-kube-controller-manager installer-1-test1-master-0 0/1 Completed 0 1h
openshift-kube-controller-manager installer-2-test1-master-0 0/1 Completed 0 24m
openshift-kube-controller-manager openshift-kube-controller-manager-test1-master-0 1/1 Running 0 24m
openshift-kube-scheduler installer-1-test1-master-0 0/1 Completed 0 1h
openshift-kube-scheduler installer-2-test1-master-0 0/1 Completed 0 24m
openshift-kube-scheduler openshift-kube-scheduler-test1-master-0 1/1 Running 0 23m
openshift-machine-config-operator machine-config-controller-6cdb889b98-s7kp2 1/1 Running 0 19m
openshift-machine-config-operator machine-config-daemon-h6x8k 1/1 Running 0 16m
openshift-machine-config-operator machine-config-daemon-tksk7 1/1 NodeLost 0 16m
openshift-machine-config-operator machine-config-operator-6c654dd6bd-fvsmz 1/1 Running 0 20m
openshift-machine-config-operator machine-config-server-q8rw8 1/1 Running 0 17m
openshift-monitoring cluster-monitoring-operator-7744c5fd65-g2sp5 0/1 Pending 0 3m
openshift-monitoring cluster-monitoring-operator-7744c5fd65-t9dc2 1/1 Unknown 0 18m
openshift-monitoring grafana-57f595895d-rmwz9 2/2 Unknown 0 55m
openshift-monitoring grafana-57f595895d-szv9p 0/2 Pending 0 3m
openshift-monitoring prometheus-k8s-0 6/6 Unknown 2 52m
openshift-monitoring prometheus-k8s-1 6/6 Unknown 1 48m
openshift-monitoring prometheus-operator-99d89bcd-d4l6h 0/1 Pending 0 3m
openshift-monitoring prometheus-operator-99d89bcd-q2bfl 1/1 Unknown 0 52m
openshift-operator-lifecycle-manager catalog-operator-9758dfdc5-4kctl 0/1 Pending 0 3m
openshift-operator-lifecycle-manager catalog-operator-9758dfdc5-sf2mj 1/1 Unknown 0 20m
openshift-operator-lifecycle-manager olm-operator-67f6b68849-48trm 1/1 Unknown 0 21m
openshift-operator-lifecycle-manager olm-operator-67f6b68849-8snt8 0/1 Pending 0 3m
openshift-operator-lifecycle-manager package-server-79fb56977f-b8sx5 0/1 Pending 0 3m
openshift-operator-lifecycle-manager package-server-79fb56977f-wh4gz 1/1 Unknown 0 20m
openshift-service-cert-signer apiservice-cabundle-injector-ccc4949d4-kxmws 1/1 Running 1 28m
openshift-service-cert-signer configmap-cabundle-injector-7598544959-cf9z9 1/1 Running 1 28m
openshift-service-cert-signer service-serving-cert-signer-785f84ccb9-jx2lk 1/1 Running 1 28m
tectonic-system kube-addon-operator-654588755f-hz65v 1/1 Running 1 2h
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
test1-master-0 Ready master 2h v1.11.0+d4cacc0
test1-worker-0-zd7hd Ready worker 2h v1.11.0+d4cacc0
[core@test1-master-0 ~]$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-member-test1-master-0 1/1 Running 0 2h
kube-system kube-flannel-8jq4j 2/2 Running 5 2h
kube-system kube-flannel-qmn2c 2/2 Running 0 2h
kube-system kube-proxy-jdzzz 1/1 Running 0 2h
kube-system kube-proxy-mfstb 1/1 Running 1 2h
kube-system kube-scheduler-qgbm4 1/1 Running 0 2h
kube-system metrics-server-5767bfc576-lfrkd 2/2 Running 0 51m
kube-system tectonic-network-operator-cwqrd 1/1 Running 1 2h
openshift-apiserver apiserver-8l9tv 1/1 Running 0 1h
openshift-cluster-api clusterapi-manager-controllers-684c97f56b-h4dk2 3/3 Running 0 1h
openshift-cluster-api machine-api-operator-c8947696b-lpd74 1/1 Running 0 1h
openshift-cluster-dns-operator cluster-dns-operator-dbb6665b5-v4wrp 1/1 Running 0 51m
openshift-cluster-dns dns-default-5wgxm 1/1 Running 1 2h
openshift-cluster-dns dns-default-wxhrb 1/1 Running 0 2h
openshift-cluster-network-operator cluster-network-operator-k2qb9 1/1 Running 0 1h
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-5f7c598d5c-jwjst 1/1 Running 0 51m
openshift-cluster-node-tuning tuned-5g6rv 1/1 Running 1 1h
openshift-cluster-node-tuning tuned-7hm86 1/1 Running 0 1h
openshift-cluster-samples-operator cluster-samples-operator-76d79b8c6c-gmkqd 1/1 Running 0 51m
openshift-cluster-version cluster-version-operator-824mk 1/1 Running 8 2h
openshift-console console-operator-7cb96f95cc-zt6p5 1/1 Running 0 51m
openshift-console openshift-console-669f887744-2qrzz 1/1 Running 0 51m
openshift-controller-manager controller-manager-qgws4 1/1 Running 0 1h
openshift-core-operators openshift-cluster-kube-apiserver-operator-75494fbfb8-pnzhs 1/1 Running 4 1h
openshift-core-operators openshift-cluster-kube-controller-manager-operator-597fcdfrw4rz 1/1 Running 4 1h
openshift-core-operators openshift-cluster-kube-scheduler-operator-657cffdbdd-5rh4b 1/1 Running 4 1h
openshift-core-operators openshift-cluster-openshift-apiserver-operator-f7546c66d-c57pl 1/1 Running 0 1h
openshift-core-operators openshift-cluster-openshift-controller-manager-operator-5fwgq25 1/1 Running 0 1h
openshift-core-operators openshift-service-cert-signer-operator-776d49f685-pwqcm 1/1 Running 1 1h
openshift-csi-operator csi-operator-5df7749cd9-vnww4 1/1 Running 0 51m
openshift-image-registry cluster-image-registry-operator-f95c59b44-d6gq5 1/1 Running 0 51m
openshift-ingress-operator ingress-operator-778475444b-789x4 1/1 Running 0 51m
openshift-ingress router-default-7q9ct 1/1 Running 1 1h
openshift-kube-apiserver installer-1-test1-master-0 0/1 Completed 0 2h
openshift-kube-apiserver installer-2-test1-master-0 0/1 Completed 0 1h
openshift-kube-apiserver openshift-kube-apiserver-test1-master-0 1/1 Running 0 1h
openshift-kube-controller-manager installer-1-test1-master-0 0/1 Completed 0 2h
openshift-kube-controller-manager installer-2-test1-master-0 0/1 Completed 0 1h
openshift-kube-controller-manager openshift-kube-controller-manager-test1-master-0 1/1 Running 0 1h
openshift-kube-scheduler installer-1-test1-master-0 0/1 Completed 0 2h
openshift-kube-scheduler installer-2-test1-master-0 0/1 Completed 0 1h
openshift-kube-scheduler openshift-kube-scheduler-test1-master-0 1/1 Running 0 1h
openshift-machine-config-operator machine-config-controller-6cdb889b98-s7kp2 1/1 Running 0 1h
openshift-machine-config-operator machine-config-daemon-h6x8k 1/1 Running 0 1h
openshift-machine-config-operator machine-config-daemon-tksk7 1/1 Running 1 1h
openshift-machine-config-operator machine-config-operator-6c654dd6bd-fvsmz 1/1 Running 0 1h
openshift-machine-config-operator machine-config-server-q8rw8 1/1 Running 0 1h
openshift-monitoring alertmanager-main-0 3/3 Running 0 43m
openshift-monitoring alertmanager-main-1 3/3 Running 0 39m
openshift-monitoring alertmanager-main-2 3/3 Running 0 39m
openshift-monitoring cluster-monitoring-operator-7744c5fd65-g2sp5 1/1 Running 0 51m
openshift-monitoring grafana-57f595895d-szv9p 2/2 Running 0 51m
openshift-monitoring kube-state-metrics-dcf7dc56d-wvfxl 3/3 Running 0 34m
openshift-monitoring node-exporter-9jgxt 2/2 Running 0 39m
openshift-monitoring node-exporter-b56s6 2/2 Running 0 39m
openshift-monitoring prometheus-k8s-0 6/6 Running 1 43m
openshift-monitoring prometheus-k8s-1 6/6 Running 1 43m
openshift-monitoring prometheus-operator-99d89bcd-d4l6h 1/1 Running 0 51m
openshift-monitoring telemeter-client-7cb686cb6-qv2hl 1/2 CrashLoopBackOff 10 32m
openshift-operator-lifecycle-manager catalog-operator-9758dfdc5-4kctl 1/1 Running 0 51m
openshift-operator-lifecycle-manager olm-operator-67f6b68849-8snt8 1/1 Running 0 51m
openshift-operator-lifecycle-manager package-server-79fb56977f-b8sx5 1/1 Running 0 51m
openshift-service-cert-signer apiservice-cabundle-injector-ccc4949d4-kxmws 1/1 Running 1 1h
openshift-service-cert-signer configmap-cabundle-injector-7598544959-cf9z9 1/1 Running 1 1h
openshift-service-cert-signer service-serving-cert-signer-785f84ccb9-jx2lk 1/1 Running 1 1h
tectonic-system kube-addon-operator-654588755f-hz65v 1/1 Running 1 2h
After reboot the master and node (4-5 mins)
[core@test1-master-0 ~]$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-member-test1-master-0 1/1 Running 1 3h
kube-system kube-flannel-8jq4j 2/2 Running 8 2h
kube-system kube-flannel-qmn2c 2/2 Running 3 3h
kube-system kube-proxy-jdzzz 1/1 Running 1 3h
kube-system kube-proxy-mfstb 1/1 Running 2 2h
kube-system kube-scheduler-qgbm4 1/1 Running 1 3h
kube-system metrics-server-5767bfc576-lfrkd 2/2 Running 2 1h
kube-system tectonic-network-operator-cwqrd 1/1 Running 2 3h
openshift-apiserver apiserver-8l9tv 1/1 Running 2 1h
openshift-cluster-api clusterapi-manager-controllers-684c97f56b-h4dk2 3/3 Running 3 1h
openshift-cluster-api machine-api-operator-c8947696b-lpd74 1/1 Running 1 1h
openshift-cluster-dns-operator cluster-dns-operator-dbb6665b5-v4wrp 1/1 Running 1 1h
openshift-cluster-dns dns-default-5wgxm 1/1 Running 2 2h
openshift-cluster-dns dns-default-wxhrb 1/1 Running 1 2h
openshift-cluster-network-operator cluster-network-operator-k2qb9 1/1 Running 1 1h
openshift-cluster-node-tuning-operator cluster-node-tuning-operator-5f7c598d5c-jwjst 1/1 Running 1 1h
openshift-cluster-node-tuning tuned-5g6rv 1/1 Running 2 1h
openshift-cluster-node-tuning tuned-7hm86 1/1 Running 1 1h
openshift-cluster-samples-operator cluster-samples-operator-76d79b8c6c-gmkqd 1/1 Running 3 1h
openshift-cluster-version cluster-version-operator-824mk 1/1 Running 9 3h
openshift-console console-operator-7cb96f95cc-zt6p5 1/1 Running 4 1h
openshift-console openshift-console-669f887744-2qrzz 1/1 Running 1 1h
openshift-controller-manager controller-manager-qgws4 1/1 Running 1 1h
openshift-core-operators openshift-cluster-kube-apiserver-operator-75494fbfb8-pnzhs 1/1 Running 5 1h
openshift-core-operators openshift-cluster-kube-controller-manager-operator-597fcdfrw4rz 1/1 Running 5 1h
openshift-core-operators openshift-cluster-kube-scheduler-operator-657cffdbdd-5rh4b 1/1 Running 5 1h
openshift-core-operators openshift-cluster-openshift-apiserver-operator-f7546c66d-c57pl 1/1 Running 1 1h
openshift-core-operators openshift-cluster-openshift-controller-manager-operator-5fwgq25 1/1 Running 1 1h
openshift-core-operators openshift-service-cert-signer-operator-776d49f685-pwqcm 1/1 Running 2 1h
openshift-csi-operator csi-operator-5df7749cd9-vnww4 1/1 Running 1 1h
openshift-image-registry cluster-image-registry-operator-f95c59b44-d6gq5 1/1 Running 4 1h
openshift-ingress-operator ingress-operator-778475444b-789x4 1/1 Running 1 1h
openshift-ingress router-default-7q9ct 1/1 Running 2 2h
openshift-kube-apiserver installer-1-test1-master-0 0/1 Completed 0 2h
openshift-kube-apiserver installer-2-test1-master-0 0/1 Completed 0 1h
openshift-kube-apiserver openshift-kube-apiserver-test1-master-0 1/1 Running 1 1h
openshift-kube-controller-manager installer-1-test1-master-0 0/1 Completed 0 2h
openshift-kube-controller-manager installer-2-test1-master-0 0/1 Completed 0 1h
openshift-kube-controller-manager openshift-kube-controller-manager-test1-master-0 1/1 Running 1 1h
openshift-kube-scheduler installer-1-test1-master-0 0/1 Completed 0 2h
openshift-kube-scheduler installer-2-test1-master-0 0/1 Completed 0 1h
openshift-kube-scheduler openshift-kube-scheduler-test1-master-0 1/1 Running 1 1h
openshift-machine-config-operator machine-config-controller-6cdb889b98-s7kp2 1/1 Running 1 1h
openshift-machine-config-operator machine-config-daemon-h6x8k 1/1 Running 1 1h
openshift-machine-config-operator machine-config-daemon-tksk7 1/1 Running 2 1h
openshift-machine-config-operator machine-config-operator-6c654dd6bd-fvsmz 1/1 Running 1 1h
openshift-machine-config-operator machine-config-server-q8rw8 1/1 Running 1 1h
openshift-monitoring alertmanager-main-0 3/3 Running 3 1h
openshift-monitoring alertmanager-main-1 3/3 Running 3 1h
openshift-monitoring alertmanager-main-2 3/3 Running 3 1h
openshift-monitoring cluster-monitoring-operator-7744c5fd65-g2sp5 1/1 Running 1 1h
openshift-monitoring grafana-57f595895d-szv9p 2/2 Running 2 1h
openshift-monitoring kube-state-metrics-dcf7dc56d-wvfxl 3/3 Running 3 55m
openshift-monitoring node-exporter-9jgxt 2/2 Running 2 1h
openshift-monitoring node-exporter-b56s6 2/2 Running 2 1h
openshift-monitoring prometheus-k8s-0 6/6 Running 7 1h
openshift-monitoring prometheus-k8s-1 6/6 Running 7 1h
openshift-monitoring prometheus-operator-99d89bcd-d4l6h 1/1 Running 1 1h
openshift-monitoring telemeter-client-7cb686cb6-qv2hl 1/2 CrashLoopBackOff 19 53m
openshift-operator-lifecycle-manager catalog-operator-9758dfdc5-4kctl 1/1 Running 1 1h
openshift-operator-lifecycle-manager olm-operator-67f6b68849-8snt8 1/1 Running 1 1h
openshift-operator-lifecycle-manager package-server-79fb56977f-b8sx5 1/1 Running 1 1h
openshift-service-cert-signer apiservice-cabundle-injector-ccc4949d4-kxmws 1/1 Running 2 1h
openshift-service-cert-signer configmap-cabundle-injector-7598544959-cf9z9 1/1 Running 2 1h
openshift-service-cert-signer service-serving-cert-signer-785f84ccb9-jx2lk 1/1 Running 2 1h
tectonic-system kube-addon-operator-654588755f-hz65v 1/1 Running 2 3h
$ kubectl logs telemeter-client-7cb686cb6-qv2hl -c telemeter-client -n openshift-monitoring
Error: failed to configure Telemeter client: both anonymize-salt and anonymize-labels must be either specified or empty
$ kubectl logs telemeter-client-7cb686cb6-qv2hl -c kube-rbac-proxy -n openshift-monitoring
2018/11/09 08:08:01 http: TLS handshake error from 10.2.1.61:35178: remote error: tls: bad certificate
2018/11/09 08:08:10 http: TLS handshake error from 10.2.1.62:43592: remote error: tls: bad certificate
2018/11/09 08:08:31 http: TLS handshake error from 10.2.1.61:35482: remote error: tls: bad certificate
2018/11/09 08:08:40 http: TLS handshake error from 10.2.1.62:43894: remote error: tls: bad certificate
$ kubectl describe pod telemeter-client-7cb686cb6-qv2hl -n openshift-monitoring
Name: telemeter-client-7cb686cb6-qv2hl
Namespace: openshift-monitoring
Priority: 0
PriorityClassName: <none>
Node: test1-worker-0-zd7hd/192.168.126.51
Start Time: Fri, 09 Nov 2018 07:27:59 +0000
Labels: k8s-app=telemeter-client
pod-template-hash=376242762
Annotations: openshift.io/scc=restricted
Status: Running
IP: 10.2.1.72
Controlled By: ReplicaSet/telemeter-client-7cb686cb6
Containers:
telemeter-client:
Container ID: cri-o://02ff26166b21a87473253f2657b7ede3e45e7893fbcfe86befd73b8be7a523a1
Image: quay.io/openshift/origin-telemeter:v4.0
Image ID: quay.io/openshift/origin-telemeter@sha256:365587a6c323cedfbc542b93ceec2a3b4dff485c82ff47f9b9a37910e0e7ab6d
Port: 8080/TCP
Host Port: 0/TCP
Command:
/usr/bin/telemeter-client
--id=$(ID)
--from=https://prometheus-k8s.openshift-monitoring.svc:9091
--from-ca-file=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
--from-token-file=/var/run/secrets/kubernetes.io/serviceaccount/token
--to=$(TO)
--to-token-file=/etc/telemeter/token
--listen=localhost:8080
--match-file=/etc/telemeter/match-rules
--anonymize-salt-file=/etc/telemeter/salt
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 09 Nov 2018 08:32:14 +0000
Finished: Fri, 09 Nov 2018 08:32:14 +0000
Ready: False
Restart Count: 21
Environment:
ID: <set to the key 'id' in secret 'telemeter-client'> Optional: false
TO: <set to the key 'to' in secret 'telemeter-client'> Optional: false
Mounts:
/etc/telemeter from secret-telemeter-client (rw)
/var/run/secrets/kubernetes.io/serviceaccount from telemeter-client-token-txvrq (ro)
kube-rbac-proxy:
Container ID: cri-o://a061c0e06c4650a5f7e0648ca584236409e0cfb8dace5830b0ad4bbda1559a3c
Image: quay.io/coreos/kube-rbac-proxy:v0.4.0
Image ID: quay.io/coreos/kube-rbac-proxy@sha256:4e803a78d802cbd5a4014861ce1e1da343c3e8a1e86577fd9c49f359472b9bd5
Port: 8443/TCP
Host Port: 0/TCP
Args:
--secure-listen-address=:8443
--upstream=http://127.0.0.1:8080/
--tls-cert-file=/etc/tls/private/tls.crt
--tls-private-key-file=/etc/tls/private/tls.key
State: Running
Started: Fri, 09 Nov 2018 08:06:10 +0000
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Fri, 09 Nov 2018 07:29:12 +0000
Finished: Fri, 09 Nov 2018 08:05:29 +0000
Ready: True
Restart Count: 1
Limits:
cpu: 20m
memory: 40Mi
Requests:
cpu: 10m
memory: 20Mi
Environment: <none>
Mounts:
/etc/tls/private from telemeter-client-tls (rw)
/var/run/secrets/kubernetes.io/serviceaccount from telemeter-client-token-txvrq (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
secret-telemeter-client:
Type: Secret (a volume populated by a Secret)
SecretName: telemeter-client
Optional: false
telemeter-client-tls:
Type: Secret (a volume populated by a Secret)
SecretName: telemeter-client-tls
Optional: false
telemeter-client-token-txvrq:
Type: Secret (a volume populated by a Secret)
SecretName: telemeter-client-token-txvrq
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 1h default-scheduler Successfully assigned openshift-monitoring/telemeter-client-7cb686cb6-qv2hl to test1-worker-0-zd7hd
Normal Pulling 1h kubelet, test1-worker-0-zd7hd pulling image "quay.io/openshift/origin-telemeter:v4.0"
Normal Pulled 1h kubelet, test1-worker-0-zd7hd Successfully pulled image "quay.io/openshift/origin-telemeter:v4.0"
Normal Pulled 1h kubelet, test1-worker-0-zd7hd Container image "quay.io/coreos/kube-rbac-proxy:v0.4.0" already present on machine
Normal Created 1h kubelet, test1-worker-0-zd7hd Created container
Normal Started 1h kubelet, test1-worker-0-zd7hd Started container
Normal Created 1h (x4 over 1h) kubelet, test1-worker-0-zd7hd Created container
Normal Started 1h (x4 over 1h) kubelet, test1-worker-0-zd7hd Started container
Normal Pulled 1h (x4 over 1h) kubelet, test1-worker-0-zd7hd Container image "quay.io/openshift/origin-telemeter:v4.0" already present on machine
Warning BackOff 31m (x155 over 1h) kubelet, test1-worker-0-zd7hd Back-off restarting failed container
Warning FailedCreatePodSandBox 29m kubelet, test1-worker-0-zd7hd Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_telemeter-client-7cb686cb6-qv2hl_openshift-monitoring_fc420c2f-e3f0-11e8-9c51-b6be4d33e3b4_1(e0f2e1bed34363d239e587ffe44ee6eb175ced9315a9bce5ff908a542d647bfd): open /run/flannel/subnet.env: no such file or directory
Normal SandboxChanged 29m (x2 over 29m) kubelet, test1-worker-0-zd7hd Pod sandbox changed, it will be killed and re-created.
Normal Pulled 28m kubelet, test1-worker-0-zd7hd Container image "quay.io/coreos/kube-rbac-proxy:v0.4.0" already present on machine
Normal Created 28m kubelet, test1-worker-0-zd7hd Created container
Normal Started 28m kubelet, test1-worker-0-zd7hd Started container
Normal Started 28m (x3 over 28m) kubelet, test1-worker-0-zd7hd Started container
Normal Pulled 27m (x4 over 28m) kubelet, test1-worker-0-zd7hd Container image "quay.io/openshift/origin-telemeter:v4.0" already present on machine
Normal Created 27m (x4 over 28m) kubelet, test1-worker-0-zd7hd Created container
Warning BackOff 2m (x120 over 28m) kubelet, test1-worker-0-zd7hd Back-off restarting failed container
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment