Skip to content

Instantly share code, notes, and snippets.

@timroster

timroster/systemctl_kubelet_out.txt

Created January 26, 2021 20:38
Show Gist options
  • Save timroster/b043cd365b168646a6351ba9a5cb51a8 to your computer and use it in GitHub Desktop.
Save timroster/b043cd365b168646a6351ba9a5cb51a8 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
systemctl_kubelet_out.txt
kubelet.service - Kubernetes Kubelet
Loaded: loaded (/etc/systemd/system/kubelet.service; disabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-mco-default-env.conf
Active: active (running) since Tue 2021-01-26 14:46:46 UTC; 5h 46min ago
Process: 2989 ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state (code=exited, status=0/SUCCESS)
Process: 2987 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
Main PID: 2991 (kubelet)
Tasks: 146 (limit: 287245)
Memory: 463.1M
CPU: 3h 58min 55.415s
CGroup: /system.slice/kubelet.service
└─2991 kubelet --node-ip=192.168.126.11 --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --kubeconfig=/var/lib/kubelet/kubeconfig --container-runtime=remote --container-runtime->
Jan 26 20:33:33 crc-lf65c-master-0 hyperkube[2991]: E0126 20:33:33.591560 2991 kubelet_pods.go:1250] Failed killing the pod "installer-4-crc-lf65c-master-0": failed to "KillPodSandbox" for "3b66dc02-2b22-41fd-af51-e9fbc41984>
Jan 26 20:33:33 crc-lf65c-master-0 hyperkube[2991]: E0126 20:33:33.595201 2991 remote_runtime.go:140] StopPodSandbox "7f60cb2130b13c29990417825319233f5673ee335313055e14f1bc701768232d" from runtime service failed: rpc error: >
Jan 26 20:33:33 crc-lf65c-master-0 hyperkube[2991]: E0126 20:33:33.595425 2991 kuberuntime_manager.go:909] Failed to stop sandbox {"cri-o" "7f60cb2130b13c29990417825319233f5673ee335313055e14f1bc701768232d"}
Jan 26 20:33:33 crc-lf65c-master-0 hyperkube[2991]: E0126 20:33:33.595508 2991 kubelet_pods.go:1250] Failed killing the pod "community-operators-q28xc": failed to "KillPodSandbox" for "05daf462-71f5-4cca-8963-2c2ec8c8113c" w>
Jan 26 20:33:33 crc-lf65c-master-0 hyperkube[2991]: E0126 20:33:33.855752 2991 remote_runtime.go:113] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_commu>
Jan 26 20:33:33 crc-lf65c-master-0 hyperkube[2991]: E0126 20:33:33.855831 2991 kuberuntime_sandbox.go:70] CreatePodSandbox for pod "community-operators-vtkqg_openshift-marketplace(dc038eb5-90bf-4917-b7ad-ed5d909b259f)" faile>
Jan 26 20:33:33 crc-lf65c-master-0 hyperkube[2991]: E0126 20:33:33.855851 2991 kuberuntime_manager.go:741] createPodSandbox for pod "community-operators-vtkqg_openshift-marketplace(dc038eb5-90bf-4917-b7ad-ed5d909b259f)" fail>
Jan 26 20:33:33 crc-lf65c-master-0 hyperkube[2991]: E0126 20:33:33.855920 2991 pod_workers.go:191] Error syncing pod dc038eb5-90bf-4917-b7ad-ed5d909b259f ("community-operators-vtkqg_openshift-marketplace(dc038eb5-90bf-4917-b>
Jan 26 20:33:33 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:33.856029 2991 event.go:291] "Event occurred" object="openshift-marketplace/community-operators-vtkqg" kind="Pod" apiVersion="v1" type="Warning" reason="FailedC>
Jan 26 20:33:33 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:33.935984 2991 worker.go:215] Non-running container probed: apiserver-658b78c545-c7c5x_openshift-oauth-apiserver(9e1418e7-979a-4517-b168-603f7ad858d0) - oauth-a>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.229481 2991 worker.go:215] Non-running container probed: apiserver-658b78c545-c7c5x_openshift-oauth-apiserver(9e1418e7-979a-4517-b168-603f7ad858d0) - oauth-a>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.284310 2991 prober.go:126] Readiness probe for "image-registry-688f586b9-k9wkk_openshift-image-registry(37b8ef2c-dfd1-4fa1-82e0-02dba3903319):registry" succe>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.374012 2991 kubelet_pods.go:1486] Generating status for "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(a2ad9b69-eb57-41d5-b090-83fc8da8a7e2)"
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.375321 2991 status_manager.go:429] Ignoring same status for pod "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(a2ad9b69-eb57-41d5-b090-83fc8da8a>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: W0126 20:30:08.059000 18 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://192.168.130.11:2379 <nil> 0 <nil>}. Err :connection error: desc>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: W0126 20:30:08.544711 18 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://localhost:2379 <nil> 0 <nil>}. Err :connection error: desc = "t>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: W0126 20:30:13.582624 18 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://192.168.130.11:2379 <nil> 0 <nil>}. Err :connection error: desc>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: W0126 20:30:14.290903 18 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://localhost:2379 <nil> 0 <nil>}. Err :connection error: desc = "t>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: W0126 20:30:14.353802 18 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://192.168.130.11:2379 <nil> 0 <nil>}. Err :connection error: desc>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: W0126 20:30:15.158542 18 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://localhost:2379 <nil> 0 <nil>}. Err :connection error: desc = "t>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: Error: context deadline exceeded
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:30:17.869766 1 main.go:198] Termination finished with exit code 1
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:30:17.869885 1 main.go:151] Deleting termination lock file "/var/log/kube-apiserver/.terminating"
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: ,StartedAt:2021-01-26 20:29:56 +0000 UTC,FinishedAt:2021-01-26 20:30:17 +0000 UTC,ContainerID:cri-o://eb4b95cf6ff58221e02c7604d69ce0627ec09a31b545d35ccd515e638aa50213,}} Ready>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: E0126 20:21:40.437910 1 reflector.go:127] k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https:>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: E0126 20:21:59.673712 1 reflector.go:127] k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://loca>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: E0126 20:22:13.536686 1 reflector.go:127] k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https:>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: E0126 20:22:52.751496 1 reflector.go:127] k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://loca>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: E0126 20:23:09.157408 1 reflector.go:127] k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https:>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: E0126 20:23:25.336008 1 reflector.go:127] k8s.io/[email protected]/tools/cache/reflector.go:156: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "https://loca>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: F0126 20:23:42.699627 1 base_controller.go:95] unable to sync caches for CertSyncController
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: ,StartedAt:2021-01-26 20:13:42 +0000 UTC,FinishedAt:2021-01-26 20:23:42 +0000 UTC,ContainerID:cri-o://71c3cf2f7ce143032b055073216275d6907cc254109ce8efd83e9c39d862b3ad,}} Ready>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: goroutine 1116 [select]:
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000d66c00)
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/delaying_queue.go:231 +0x405
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: created by k8s.io/client-go/util/workqueue.newDelayingQueue
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/delaying_queue.go:68 +0x185
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: goroutine 1146 [chan receive]:
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc001110ea0)
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/queue.go:198 +0xac
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: created by k8s.io/client-go/util/workqueue.newQueue
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/queue.go:58 +0x135
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: goroutine 1148 [select]:
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc001111200)
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/delaying_queue.go:231 +0x405
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: created by k8s.io/client-go/util/workqueue.newDelayingQueue
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/delaying_queue.go:68 +0x185
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: goroutine 1155 [chan receive]:
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc001111980)
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/queue.go:198 +0xac
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: created by k8s.io/client-go/util/workqueue.newQueue
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/queue.go:58 +0x135
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: goroutine 1157 [select]:
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc001111ec0)
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/delaying_queue.go:231 +0x405
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: created by k8s.io/client-go/util/workqueue.newDelayingQueue
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/delaying_queue.go:68 +0x185
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: goroutine 1164 [chan receive]:
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc000ce6240)
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/queue.go:198 +0xac
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: created by k8s.io/client-go/util/workqueue.newQueue
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/queue.go:58 +0x135
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: goroutine 1166 [select]:
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/client-go/util/workqueue.(*delayingType).waitingLoop(0xc000ce6420)
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/delaying_queue.go:231 +0x405
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: created by k8s.io/client-go/util/workqueue.newDelayingQueue
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: k8s.io/[email protected]/util/workqueue/delaying_queue.go:68 +0x185
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: ,StartedAt:2021-01-26 20:32:13 +0000 UTC,FinishedAt:2021-01-26 20:32:45 +0000 UTC,ContainerID:cri-o://3a8e6296db7808617ae4945842797f989bed88a268ca466618e19de316bdda98,}} Ready>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.376612 2991 kubelet_pods.go:1486] Generating status for "multus-6n5tj_openshift-multus(fb4e2cbd-6afc-48f0-9219-6c9380e2d00a)"
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.377763 2991 status_manager.go:429] Ignoring same status for pod "multus-6n5tj_openshift-multus(fb4e2cbd-6afc-48f0-9219-6c9380e2d00a)", status: {Phase:Running>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.378321 2991 volume_manager.go:372] Waiting for volumes to attach and mount for pod "multus-6n5tj_openshift-multus(fb4e2cbd-6afc-48f0-9219-6c9380e2d00a)"
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.378593 2991 volume_manager.go:403] All volumes are attached and mounted for pod "multus-6n5tj_openshift-multus(fb4e2cbd-6afc-48f0-9219-6c9380e2d00a)"
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.378863 2991 kuberuntime_manager.go:664] computePodActions got {KillPod:false CreateSandbox:false SandboxID:97bdbe29ee375c53afce837ffbd2f6d85ba00eb5cdf0a66bd2>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.378247 2991 volume_manager.go:372] Waiting for volumes to attach and mount for pod "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(a2ad9b69-eb57->
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.380008 2991 volume_manager.go:403] All volumes are attached and mounted for pod "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(a2ad9b69-eb57-41d>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.380498 2991 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: eb4b95cf6ff58221e02c7604d69ce0627ec09a31b545d35ccd515e638aa50213
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.380770 2991 kuberuntime_manager.go:593] Container {Name:kube-apiserver Image:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4285931a08aedfb838854f41>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: echo "Copying system trust bundle"
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: cp -f /etc/kubernetes/static-pod-certs/configmaps/trusted-ca-bundle/ca-bundle.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: fi
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: echo -n "Waiting for port :6443 to be released."
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: tries=0
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: while [ -n "$(ss -Htan '( sport = 6443 )')" ]; do
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: echo -n "."
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: sleep 1
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: (( tries += 1 ))
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: if [[ "${tries}" -gt 105 ]]; then
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: echo "timed out waiting for port :6443 to be released"
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: exit 1
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: fi
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: done
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: echo
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: exec watch-termination --termination-touch-file=/var/log/kube-apiserver/.terminating --termination-log-file=/var/log/kube-apiserver/termination.log --kubeconfig=/etc/kubernete>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: ] WorkingDir: Ports:[{Name: HostPort:6443 ContainerPort:6443 Protocol:TCP HostIP:}] EnvFrom:[] Env:[{Name:POD_NAME Value: ValueFrom:&EnvVarSource{FieldRef:&ObjectFieldSelector>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.385829 2991 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 3a8e6296db7808617ae4945842797f989bed88a268ca466618e19de316bdda98
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.386926 2991 kuberuntime_manager.go:593] Container {Name:kube-apiserver-check-endpoints Image:quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c3c6f37d1>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.387522 2991 kuberuntime_manager.go:664] computePodActions got {KillPod:false CreateSandbox:false SandboxID:7381656b9cd9bcb29c702bbf2274d13b8ab32dd5bc20db8244>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.388450 2991 kuberuntime_manager.go:867] checking backoff for container "kube-apiserver" in pod "kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(a2>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.408428 2991 kuberuntime_manager.go:877] back-off 5m0s restarting failed container=kube-apiserver pod=kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserv>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.408470 2991 kuberuntime_manager.go:867] checking backoff for container "kube-apiserver-check-endpoints" in pod "kube-apiserver-crc-lf65c-master-0_openshift-k>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.408896 2991 event.go:291] "Event occurred" object="openshift-kube-apiserver/kube-apiserver-crc-lf65c-master-0" kind="Pod" apiVersion="v1" type="Warning" reas>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.410559 2991 kuberuntime_manager.go:877] back-off 5m0s restarting failed container=kube-apiserver-check-endpoints pod=kube-apiserver-crc-lf65c-master-0_opensh>
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: E0126 20:33:34.410640 2991 pod_workers.go:191] Error syncing pod a2ad9b69-eb57-41d5-b090-83fc8da8a7e2 ("kube-apiserver-crc-lf65c-master-0_openshift-kube-apiserver(a2ad9b69->
Jan 26 20:33:34 crc-lf65c-master-0 hyperkube[2991]: I0126 20:33:34.410855 2991 event.go:291] "Event occurred" object="openshift-kube-apiserver/kube-apiserver-crc-lf65c-master-0" kind="Pod" apiVersion="v1" type="Warning" reas>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment