Skip to content

Instantly share code, notes, and snippets.

@FilBot3
Last active October 29, 2021 03:07
Show Gist options
  • Select an option

  • Save FilBot3/e8709ef7809527075bb5dc1df7288782 to your computer and use it in GitHub Desktop.

Select an option

Save FilBot3/e8709ef7809527075bb5dc1df7288782 to your computer and use it in GitHub Desktop.
k3s-ansible failing to connect workers and master failing to start on Ubuntu Server 21.10 arm64.
Oct 29 03:01:20 ubuntu-k3s-master systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Oct 29 03:01:20 ubuntu-k3s-master systemd[1]: k3s.service: Failed with result 'exit-code'.
Oct 29 03:01:20 ubuntu-k3s-master systemd[1]: k3s.service: Consumed 32.463s CPU time.
Oct 29 03:01:26 ubuntu-k3s-master systemd[1]: k3s.service: Scheduled restart job, restart counter is at 43.
Oct 29 03:01:26 ubuntu-k3s-master systemd[1]: Stopped Lightweight Kubernetes.
Oct 29 03:01:26 ubuntu-k3s-master systemd[1]: k3s.service: Consumed 32.463s CPU time.
Oct 29 03:01:26 ubuntu-k3s-master systemd[1]: Starting Lightweight Kubernetes...
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:26.844427808Z" level=info msg="Starting k3s v1.21.5+k3s2 (724ef700)"
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:26.845155773Z" level=info msg="Cluster bootstrap already complete"
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:26.910591057Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:26.910806645Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:26.911343818Z" level=info msg="Database tables and indexes are up to date"
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:26.929490456Z" level=info msg="Kine listening on unix://kine.sock"
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:26.930228940Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: I1029 03:01:26.934107 20645 server.go:656] external host was not specified, using 192.168.86.90
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: I1029 03:01:26.935106 20645 server.go:195] Version: v1.21.5+k3s2
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: I1029 03:01:26.947081 20645 shared_informer.go:240] Waiting for caches to sync for node_authorizer
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: I1029 03:01:26.951810 20645 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: I1029 03:01:26.951909 20645 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: I1029 03:01:26.957699 20645 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Oct 29 03:01:26 ubuntu-k3s-master k3s[20645]: I1029 03:01:26.957818 20645 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
Oct 29 03:01:27 ubuntu-k3s-master k3s[20645]: I1029 03:01:27.047492 20645 instance.go:283] Using reconciler: lease
Oct 29 03:01:27 ubuntu-k3s-master k3s[20645]: I1029 03:01:27.210896 20645 rest.go:130] the default service ipfamily for this cluster is: IPv4
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: W1029 03:01:28.306166 20645 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: W1029 03:01:28.353592 20645 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: W1029 03:01:28.370654 20645 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: W1029 03:01:28.399720 20645 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: W1029 03:01:28.411878 20645 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: W1029 03:01:28.439420 20645 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: W1029 03:01:28.439518 20645 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: I1029 03:01:28.479852 20645 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: I1029 03:01:28.479949 20645 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.525238764Z" level=info msg="Waiting for API server to become available"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.525265764Z" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.527631638Z" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.529529133Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m0s --port=0 --profiling=false"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.533198405Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.533333957Z" level=info msg="To join node to cluster: k3s agent -s https://192.168.86.90:6443 -t ${NODE_TOKEN}"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.537082746Z" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.537215817Z" level=info msg="Run: k3s kubectl"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.617610131Z" level=info msg="Cluster-Http-Server 2021/10/29 03:01:28 http: TLS handshake error from 192.168.86.93:50518: remote error: tls: bad certificate"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.641702844Z" level=info msg="Cluster-Http-Server 2021/10/29 03:01:28 http: TLS handshake error from 127.0.0.1:46448: remote error: tls: bad certificate"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.661967215Z" level=info msg="Cluster-Http-Server 2021/10/29 03:01:28 http: TLS handshake error from 127.0.0.1:46454: remote error: tls: bad certificate"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.719043521Z" level=error msg="runtime core not ready"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.724498402Z" level=info msg="certificate CN=ubuntu-k3s-master signed by CN=k3s-server-ca@1635475569: notBefore=2021-10-29 02:46:09 +0000 UTC notAfter=2022-10-29 03:01:28 +0000 UTC"
Oct 29 03:01:28 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:28.744091047Z" level=info msg="certificate CN=system:node:ubuntu-k3s-master,O=system:nodes signed by CN=k3s-client-ca@1635475569: notBefore=2021-10-29 02:46:09 +0000 UTC notAfter=2022-10-29 03:01:28 +0000 UTC"
Oct 29 03:01:29 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:29.065249778Z" level=info msg="Module overlay was already loaded"
Oct 29 03:01:29 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:29.065383164Z" level=info msg="Module nf_conntrack was already loaded"
Oct 29 03:01:29 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:29.065435811Z" level=info msg="Module br_netfilter was already loaded"
Oct 29 03:01:29 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:29.065485958Z" level=info msg="Module iptable_nat was already loaded"
Oct 29 03:01:29 ubuntu-k3s-master k3s[20645]: W1029 03:01:29.065602 20645 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Oct 29 03:01:29 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:29.072873019Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Oct 29 03:01:29 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:29.073516283Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:30.077459705Z" level=info msg="Containerd is now running"
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:30.108517454Z" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:30.118504625Z" level=info msg="Handling backend connection request [ubuntu-k3s-master]"
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:30.120927609Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/5b1d238ed00ea9bbae6aa96d867c1264a96ad48b42e55d56570d54d05c8d8a7a/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ubuntu-k3s-master --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:30.122806346Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=ubuntu-k3s-master --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: W1029 03:01:30.123682 20645 server.go:220] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: I1029 03:01:30.190560 20645 server.go:436] "Kubelet version" kubeletVersion="v1.21.5+k3s2"
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: E1029 03:01:30.207504 20645 node.go:161] Failed to retrieve node info: nodes "ubuntu-k3s-master" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: I1029 03:01:30.287887 20645 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: W1029 03:01:30.288013 20645 manager.go:159] Cannot detect current cgroup on cgroup v2
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:30.801576492Z" level=info msg="Handling backend connection request [ubuntu-k3s-wkr2]"
Oct 29 03:01:30 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:30.802406789Z" level=info msg="Handling backend connection request [ubuntu-k3s-wkr1]"
Oct 29 03:01:31 ubuntu-k3s-master k3s[20645]: E1029 03:01:31.401210 20645 node.go:161] Failed to retrieve node info: nodes "ubuntu-k3s-master" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Oct 29 03:01:33 ubuntu-k3s-master k3s[20645]: E1029 03:01:33.514817 20645 node.go:161] Failed to retrieve node info: nodes "ubuntu-k3s-master" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Oct 29 03:01:33 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:33.735463394Z" level=info msg="Cluster-Http-Server 2021/10/29 03:01:33 http: TLS handshake error from 192.168.86.93:50538: remote error: tls: bad certificate"
Oct 29 03:01:33 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:33.820412072Z" level=error msg="runtime core not ready"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:35.112561057Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: W1029 03:01:35.329555 20645 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.349572 20645 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.351028 20645 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.351662 20645 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.351864 20645 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.352189 20645 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.352279 20645 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.354402 20645 kubelet.go:404] "Attempting to sync node with API server"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.354616 20645 kubelet.go:272] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.354729 20645 kubelet.go:283] "Adding apiserver pod source"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.354800 20645 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.363116 20645 kuberuntime_manager.go:222] "Container runtime initialized" containerRuntime="containerd" version="v1.4.11-k3s1" apiVersion="v1alpha2"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.365317 20645 server.go:1191] "Started kubelet"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.378464 20645 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.378536 20645 server.go:149] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.381331 20645 server.go:409] "Adding debug handlers to kubelet server"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.390235 20645 volume_manager.go:271] "Starting Kubelet Volume Manager"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.391563 20645 desired_state_of_world_populator.go:141] "Desired state populator starts to run"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: E1029 03:01:35.413832 20645 cri_stats_provider.go:369] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: E1029 03:01:35.413984 20645 kubelet.go:1306] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.493875 20645 kuberuntime_manager.go:1044] "Updating runtime config through cri with podcidr" CIDR="10.42.0.0/24"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.495656 20645 kubelet_network.go:76] "Updating Pod CIDR" originalPodCIDR="" newPodCIDR="10.42.0.0/24"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.500792 20645 kubelet_node_status.go:71] "Attempting to register node" node="ubuntu-k3s-master"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.563958 20645 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv4
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.675192 20645 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv6
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.675309 20645 status_manager.go:157] "Starting to sync pod status with apiserver"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: I1029 03:01:35.676025 20645 kubelet.go:1846] "Starting kubelet main sync loop"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: E1029 03:01:35.676239 20645 kubelet.go:1870] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: E1029 03:01:35.777315 20645 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 29 03:01:35 ubuntu-k3s-master k3s[20645]: E1029 03:01:35.977605 20645 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: E1029 03:01:36.378718 20645 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.385007 20645 apiserver.go:52] "Watching apiserver"
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.647169 20645 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.647176 20645 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.648230 20645 dynamic_serving_content.go:130] Starting serving-cert::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt::/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.649185 20645 secure_serving.go:202] Serving securely on 127.0.0.1:6444
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.649935 20645 tlsconfig.go:240] Starting DynamicServingCertificateController
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.650762 20645 dynamic_serving_content.go:130] Starting aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.651610 20645 apiservice_controller.go:97] Starting APIServiceRegistrationController
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.652260 20645 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.652806 20645 available_controller.go:475] Starting AvailableConditionController
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.653363 20645 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.654100 20645 autoregister_controller.go:141] Starting autoregister controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.654653 20645 cache.go:32] Waiting for caches to sync for autoregister controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.656068 20645 controller.go:83] Starting OpenAPI AggregationController
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.650774 20645 apf_controller.go:294] Starting API Priority and Fairness config controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.660232 20645 crdregistration_controller.go:111] Starting crd-autoregister controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.660987 20645 shared_informer.go:240] Waiting for caches to sync for crd-autoregister
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.675533 20645 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/server/tls/client-ca.crt
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.675667 20645 dynamic_cafile_content.go:167] Starting request-header::/var/lib/rancher/k3s/server/tls/request-header-ca.crt
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.675745 20645 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.675774 20645 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.650867 20645 customresource_discovery_controller.go:209] Starting DiscoveryController
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.687643 20645 controller.go:86] Starting OpenAPI controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.687748 20645 naming_controller.go:291] Starting NamingConditionController
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.687805 20645 establishing_controller.go:76] Starting EstablishingController
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.687855 20645 nonstructuralschema_controller.go:192] Starting NonStructuralSchemaConditionController
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.687902 20645 apiapproval_controller.go:186] Starting KubernetesAPIApprovalPolicyConformantConditionController
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.687996 20645 crd_finalizer.go:266] Starting CRDFinalizer
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.848229 20645 shared_informer.go:247] Caches are synced for node_authorizer
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.863554 20645 controller.go:611] quota admission added evaluator for: leases.coordination.k8s.io
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.875711 20645 trace.go:205] Trace[1833545207]: "GuaranteedUpdate etcd3" type:*coordination.Lease (29-Oct-2021 03:01:29.169) (total time: 7706ms):
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[1833545207]: ---"Transaction prepared" 7696ms (03:01:00.865)
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[1833545207]: [7.706377938s] [7.706377938s] END
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.878456 20645 trace.go:205] Trace[1686096278]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1,user-agent:k3s/v1.21.5+k3s2 (linux/arm64) kubernetes/724ef70,client:192.168.86.91,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/1.1 (29-Oct-2021 03:01:29.168) (total time: 7709ms):
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[1686096278]: ---"Object stored in database" 7708ms (03:01:00.878)
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[1686096278]: [7.709408595s] [7.709408595s] END
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.877879 20645 shared_informer.go:247] Caches are synced for cluster_authentication_trust_controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.885163 20645 apf_controller.go:299] Running API Priority and Fairness config worker
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.891497 20645 trace.go:205] Trace[113903439]: "GuaranteedUpdate etcd3" type:*coordination.Lease (29-Oct-2021 03:01:28.981) (total time: 7910ms):
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[113903439]: ---"Transaction prepared" 7884ms (03:01:00.865)
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[113903439]: [7.910319802s] [7.910319802s] END
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.891954 20645 trace.go:205] Trace[276779008]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr2,user-agent:k3s/v1.21.5+k3s2 (linux/arm64) kubernetes/724ef70,client:192.168.86.92,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/1.1 (29-Oct-2021 03:01:28.980) (total time: 7911ms):
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[276779008]: ---"Object stored in database" 7910ms (03:01:00.891)
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[276779008]: [7.911049268s] [7.911049268s] END
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.913031 20645 trace.go:205] Trace[338193932]: "GuaranteedUpdate etcd3" type:*coordination.Lease (29-Oct-2021 03:01:35.472) (total time: 1440ms):
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[338193932]: ---"Transaction prepared" 1412ms (03:01:00.884)
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[338193932]: [1.440830952s] [1.440830952s] END
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.913496 20645 trace.go:205] Trace[769123758]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-master,user-agent:k3s/v1.21.5+k3s2 (linux/arm64) kubernetes/724ef70,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/1.1 (29-Oct-2021 03:01:35.471) (total time: 1441ms):
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[769123758]: ---"Object stored in database" 1441ms (03:01:00.913)
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[769123758]: [1.441577677s] [1.441577677s] END
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.953140 20645 cache.go:39] Caches are synced for APIServiceRegistrationController controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.961081 20645 cache.go:39] Caches are synced for AvailableConditionController controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.963004 20645 shared_informer.go:247] Caches are synced for crd-autoregister
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.963234 20645 cache.go:39] Caches are synced for autoregister controller
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: I1029 03:01:36.980683 20645 trace.go:205] Trace[358137343]: "Create" url:/api/v1/namespaces/default/events,user-agent:k3s/v1.21.5+k3s2 (linux/arm64) kubernetes/724ef70,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/1.1 (29-Oct-2021 03:01:35.441) (total time: 1538ms):
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[358137343]: ---"Object stored in database" 1538ms (03:01:00.980)
Oct 29 03:01:36 ubuntu-k3s-master k3s[20645]: Trace[358137343]: [1.538597031s] [1.538597031s] END
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.010768 20645 trace.go:205] Trace[1292014459]: "Create" url:/api/v1/nodes,user-agent:k3s/v1.21.5+k3s2 (linux/arm64) kubernetes/724ef70,client:127.0.0.1,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/1.1 (29-Oct-2021 03:01:35.523) (total time: 1486ms):
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: Trace[1292014459]: [1.486963759s] [1.486963759s] END
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: E1029 03:01:37.021562 20645 controller.go:156] Unable to remove old endpoints from kubernetes service: no master IPs were listed in storage, refusing to erase all endpoints for the kubernetes service
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.034102 20645 kubelet_node_status.go:109] "Node was previously registered" node="ubuntu-k3s-master"
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.035308 20645 kubelet_node_status.go:74] "Successfully registered node" node="ubuntu-k3s-master"
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.066653 20645 setters.go:577] "Node became not ready" node="ubuntu-k3s-master" condition={Type:Ready Status:False LastHeartbeatTime:2021-10-29 03:01:37.066370636 +0000 UTC m=+10.946961409 LastTransitionTime:2021-10-29 03:01:37.066370636 +0000 UTC m=+10.946961409 Reason:KubeletNotReady Message:container runtime status check may not have completed yet}
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: E1029 03:01:37.181482 20645 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.696253 20645 node.go:172] Successfully retrieved node IP: 192.168.86.90
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.696360 20645 server_others.go:141] Detected node IP 192.168.86.90
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.767439 20645 server_others.go:207] kube-proxy running in dual-stack mode, IPv4-primary
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.768070 20645 server_others.go:213] Using iptables Proxier.
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.768677 20645 server_others.go:220] creating dualStackProxier for iptables.
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: W1029 03:01:37.769280 20645 server_others.go:519] detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.821482 20645 server.go:643] Version: v1.21.5+k3s2
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.823530 20645 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.826146 20645 config.go:315] Starting service config controller
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.826228 20645 shared_informer.go:240] Waiting for caches to sync for service config
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.826304 20645 config.go:224] Starting endpoint slice config controller
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.826331 20645 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.864353 20645 cpu_manager.go:199] "Starting CPU manager" policy="none"
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.864434 20645 cpu_manager.go:200] "Reconciling" reconcilePeriod="10s"
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.864506 20645 state_mem.go:36] "Initialized new in-memory state store"
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.865153 20645 state_mem.go:88] "Updated default CPUSet" cpuSet=""
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.865218 20645 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.865251 20645 policy_none.go:44] "None policy: Start"
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.869314 20645 manager.go:600] "Failed to retrieve checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.875339 20645 plugin_manager.go:114] "Starting Kubelet Plugin Manager"
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.927037 20645 shared_informer.go:247] Caches are synced for endpoint slice config
Oct 29 03:01:37 ubuntu-k3s-master k3s[20645]: I1029 03:01:37.927044 20645 shared_informer.go:247] Caches are synced for service config
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.783494 20645 topology_manager.go:187] "Topology Admit Handler"
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.786044 20645 topology_manager.go:187] "Topology Admit Handler"
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.787025 20645 topology_manager.go:187] "Topology Admit Handler"
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.787993 20645 topology_manager.go:187] "Topology Admit Handler"
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.788887 20645 topology_manager.go:187] "Topology Admit Handler"
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:38.840481311Z" level=info msg="Cluster-Http-Server 2021/10/29 03:01:38 http: TLS handshake error from 192.168.86.93:50552: remote error: tls: bad certificate"
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.849353 20645 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6tft\" (UniqueName: \"kubernetes.io/projected/9e6fcb26-4756-4ba1-86d3-3d0c01c354d7-kube-api-access-s6tft\") pod \"local-path-provisioner-5ff76fc89d-4s8wv\" (UID: \"9e6fcb26-4756-4ba1-86d3-3d0c01c354d7\") "
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.849510 20645 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"values\" (UniqueName: \"kubernetes.io/configmap/3ff1ffb5-9742-4f2c-b44f-3076ace11ce6-values\") pod \"helm-install-traefik-vb664\" (UID: \"3ff1ffb5-9742-4f2c-b44f-3076ace11ce6\") "
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.849636 20645 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"content\" (UniqueName: \"kubernetes.io/configmap/3ff1ffb5-9742-4f2c-b44f-3076ace11ce6-content\") pod \"helm-install-traefik-vb664\" (UID: \"3ff1ffb5-9742-4f2c-b44f-3076ace11ce6\") "
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.849775 20645 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/127a7a55-1134-44da-8a24-9fbe9b076616-config-volume\") pod \"coredns-7448499f4d-4gc4z\" (UID: \"127a7a55-1134-44da-8a24-9fbe9b076616\") "
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.849869 20645 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k2zt\" (UniqueName: \"kubernetes.io/projected/127a7a55-1134-44da-8a24-9fbe9b076616-kube-api-access-9k2zt\") pod \"coredns-7448499f4d-4gc4z\" (UID: \"127a7a55-1134-44da-8a24-9fbe9b076616\") "
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.849955 20645 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4b2f7c-6868-4c65-8dfc-8e9b29b57696-tmp-dir\") pod \"metrics-server-86cbb8457f-dt8lv\" (UID: \"ae4b2f7c-6868-4c65-8dfc-8e9b29b57696\") "
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.850032 20645 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6g5v\" (UniqueName: \"kubernetes.io/projected/ae4b2f7c-6868-4c65-8dfc-8e9b29b57696-kube-api-access-g6g5v\") pod \"metrics-server-86cbb8457f-dt8lv\" (UID: \"ae4b2f7c-6868-4c65-8dfc-8e9b29b57696\") "
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.850115 20645 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e6fcb26-4756-4ba1-86d3-3d0c01c354d7-config-volume\") pod \"local-path-provisioner-5ff76fc89d-4s8wv\" (UID: \"9e6fcb26-4756-4ba1-86d3-3d0c01c354d7\") "
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.850193 20645 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"values\" (UniqueName: \"kubernetes.io/configmap/1fc8d62b-7abd-4139-8971-5c3a78ccd908-values\") pod \"helm-install-traefik-crd-brjrl\" (UID: \"1fc8d62b-7abd-4139-8971-5c3a78ccd908\") "
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.850298 20645 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcvj\" (UniqueName: \"kubernetes.io/projected/1fc8d62b-7abd-4139-8971-5c3a78ccd908-kube-api-access-jdcvj\") pod \"helm-install-traefik-crd-brjrl\" (UID: \"1fc8d62b-7abd-4139-8971-5c3a78ccd908\") "
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.850389 20645 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxspx\" (UniqueName: \"kubernetes.io/projected/3ff1ffb5-9742-4f2c-b44f-3076ace11ce6-kube-api-access-wxspx\") pod \"helm-install-traefik-vb664\" (UID: \"3ff1ffb5-9742-4f2c-b44f-3076ace11ce6\") "
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.850473 20645 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"content\" (UniqueName: \"kubernetes.io/configmap/1fc8d62b-7abd-4139-8971-5c3a78ccd908-content\") pod \"helm-install-traefik-crd-brjrl\" (UID: \"1fc8d62b-7abd-4139-8971-5c3a78ccd908\") "
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: I1029 03:01:38.850513 20645 reconciler.go:157] "Reconciler: start to sync state"
Oct 29 03:01:38 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:38.928134154Z" level=error msg="runtime core not ready"
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: E1029 03:01:39.558557 20645 remote_runtime.go:116] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"9ae7b4e0b5685d2621a392cfb1d9ecd995df7dbd04abe321626df8db0eab3670\": open /run/flannel/subnet.env: no such file or directory"
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: E1029 03:01:39.558766 20645 kuberuntime_sandbox.go:68] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"9ae7b4e0b5685d2621a392cfb1d9ecd995df7dbd04abe321626df8db0eab3670\": open /run/flannel/subnet.env: no such file or directory" pod="kube-system/helm-install-traefik-vb664"
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: E1029 03:01:39.558856 20645 kuberuntime_manager.go:790] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"9ae7b4e0b5685d2621a392cfb1d9ecd995df7dbd04abe321626df8db0eab3670\": open /run/flannel/subnet.env: no such file or directory" pod="kube-system/helm-install-traefik-vb664"
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: E1029 03:01:39.559086 20645 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"helm-install-traefik-vb664_kube-system(3ff1ffb5-9742-4f2c-b44f-3076ace11ce6)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"helm-install-traefik-vb664_kube-system(3ff1ffb5-9742-4f2c-b44f-3076ace11ce6)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"9ae7b4e0b5685d2621a392cfb1d9ecd995df7dbd04abe321626df8db0eab3670\\\": open /run/flannel/subnet.env: no such file or directory\"" pod="kube-system/helm-install-traefik-vb664" podUID=3ff1ffb5-9742-4f2c-b44f-3076ace11ce6
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: I1029 03:01:39.687129 20645 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: I1029 03:01:39.687240 20645 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:39.773444395Z" level=info msg="Waiting for cloud-controller-manager privileges to become available"
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:39.775932434Z" level=info msg="Kube API server is now running"
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:39.776602624Z" level=info msg="k3s is up and running"
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: Flag --address has been deprecated, see --bind-address instead.
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: I1029 03:01:39.816277 20645 controllermanager.go:175] Version: v1.21.5+k3s2
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: I1029 03:01:39.819093 20645 deprecated_insecure_serving.go:56] Serving insecurely on 127.0.0.1:10252
Oct 29 03:01:39 ubuntu-k3s-master k3s[20645]: I1029 03:01:39.987816 20645 request.go:668] Waited for 1.030807477s due to client-side throttling, not priority and fairness, request: POST:https://127.0.0.1:6443/api/v1/namespaces/kube-system/serviceaccounts/metrics-server/token
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.033418 20645 remote_runtime.go:116] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"5d9595d890b46ac5a0152423491dc54ca46bb0c1f88ae9eea73222124c87ec44\": open /run/flannel/subnet.env: no such file or directory"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.033624 20645 kuberuntime_sandbox.go:68] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"5d9595d890b46ac5a0152423491dc54ca46bb0c1f88ae9eea73222124c87ec44\": open /run/flannel/subnet.env: no such file or directory" pod="kube-system/helm-install-traefik-crd-brjrl"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.033743 20645 kuberuntime_manager.go:790] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"5d9595d890b46ac5a0152423491dc54ca46bb0c1f88ae9eea73222124c87ec44\": open /run/flannel/subnet.env: no such file or directory" pod="kube-system/helm-install-traefik-crd-brjrl"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.033960 20645 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"helm-install-traefik-crd-brjrl_kube-system(1fc8d62b-7abd-4139-8971-5c3a78ccd908)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"helm-install-traefik-crd-brjrl_kube-system(1fc8d62b-7abd-4139-8971-5c3a78ccd908)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"5d9595d890b46ac5a0152423491dc54ca46bb0c1f88ae9eea73222124c87ec44\\\": open /run/flannel/subnet.env: no such file or directory\"" pod="kube-system/helm-install-traefik-crd-brjrl" podUID=1fc8d62b-7abd-4139-8971-5c3a78ccd908
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: W1029 03:01:40.097846 20645 authorization.go:47] Authorization is disabled
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: W1029 03:01:40.097925 20645 authentication.go:47] Authentication is disabled
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: I1029 03:01:40.097989 20645 deprecated_insecure_serving.go:54] Serving healthz insecurely on 127.0.0.1:10251
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.182371 20645 remote_runtime.go:116] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"7c838226c9cf05ed5202bf2e4b4f111dbd903bf86864e45824d1ba390efc7640\": open /run/flannel/subnet.env: no such file or directory"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.189387 20645 kuberuntime_sandbox.go:68] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"7c838226c9cf05ed5202bf2e4b4f111dbd903bf86864e45824d1ba390efc7640\": open /run/flannel/subnet.env: no such file or directory" pod="kube-system/local-path-provisioner-5ff76fc89d-4s8wv"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.189520 20645 kuberuntime_manager.go:790] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"7c838226c9cf05ed5202bf2e4b4f111dbd903bf86864e45824d1ba390efc7640\": open /run/flannel/subnet.env: no such file or directory" pod="kube-system/local-path-provisioner-5ff76fc89d-4s8wv"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.189727 20645 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"local-path-provisioner-5ff76fc89d-4s8wv_kube-system(9e6fcb26-4756-4ba1-86d3-3d0c01c354d7)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"local-path-provisioner-5ff76fc89d-4s8wv_kube-system(9e6fcb26-4756-4ba1-86d3-3d0c01c354d7)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"7c838226c9cf05ed5202bf2e4b4f111dbd903bf86864e45824d1ba390efc7640\\\": open /run/flannel/subnet.env: no such file or directory\"" pod="kube-system/local-path-provisioner-5ff76fc89d-4s8wv" podUID=9e6fcb26-4756-4ba1-86d3-3d0c01c354d7
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.230508476Z" level=info msg="Stopped tunnel to 127.0.0.1:6443"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.231599379Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.232243365Z" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:6443/v1-k3s/connect"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.236556143Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-9.18.2.tgz"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.237501123Z" level=info msg="Writing static file: /var/lib/rancher/k3s/server/static/charts/traefik-crd-9.18.2.tgz"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.239110218Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.271367307Z" level=info msg="Handling backend connection request [ubuntu-k3s-master]"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.277398086Z" level=info msg="Node CIDR assigned for: ubuntu-k3s-master"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: I1029 03:01:40.278630 20645 flannel.go:92] Determining IP address of default interface
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: I1029 03:01:40.279669 20645 flannel.go:105] Using interface with name eth0 and address 192.168.86.90
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: I1029 03:01:40.296542 20645 kube.go:116] Waiting 10m0s for node controller to sync
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: I1029 03:01:40.297445 20645 kube.go:299] Starting kube subnet manager
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.342061575Z" level=info msg="labels have already set on node: ubuntu-k3s-master"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.419824655Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/local-storage.yaml"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.420603083Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-delegator.yaml"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.421214570Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/auth-reader.yaml"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.422331935Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-apiservice.yaml"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.423083938Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-service.yaml"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.423653962Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/resource-reader.yaml"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.424379928Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/ccm.yaml"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.425059766Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/coredns.yaml"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.425944950Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/rolebindings.yaml"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.426635065Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/traefik.yaml"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.427482344Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/aggregated-metrics-reader.yaml"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.428122274Z" level=info msg="Writing manifest: /var/lib/rancher/k3s/server/manifests/metrics-server/metrics-server-deployment.yaml"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.443470 20645 remote_runtime.go:116] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"3fe26e2f5928508c20018ce3f29b7d56cb2dfcc96fb7e34126a481cc2e97abc5\": open /run/flannel/subnet.env: no such file or directory"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.443642 20645 kuberuntime_sandbox.go:68] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"3fe26e2f5928508c20018ce3f29b7d56cb2dfcc96fb7e34126a481cc2e97abc5\": open /run/flannel/subnet.env: no such file or directory" pod="kube-system/coredns-7448499f4d-4gc4z"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.443727 20645 kuberuntime_manager.go:790] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"3fe26e2f5928508c20018ce3f29b7d56cb2dfcc96fb7e34126a481cc2e97abc5\": open /run/flannel/subnet.env: no such file or directory" pod="kube-system/coredns-7448499f4d-4gc4z"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.443904 20645 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-7448499f4d-4gc4z_kube-system(127a7a55-1134-44da-8a24-9fbe9b076616)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"coredns-7448499f4d-4gc4z_kube-system(127a7a55-1134-44da-8a24-9fbe9b076616)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"3fe26e2f5928508c20018ce3f29b7d56cb2dfcc96fb7e34126a481cc2e97abc5\\\": open /run/flannel/subnet.env: no such file or directory\"" pod="kube-system/coredns-7448499f4d-4gc4z" podUID=127a7a55-1134-44da-8a24-9fbe9b076616
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: I1029 03:01:40.526575 20645 network_policy_controller.go:144] Starting network policy controller
Oct 29 03:01:40 ubuntu-k3s-master systemd[1]: Started Lightweight Kubernetes.
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.636835410Z" level=info msg="Starting k3s.cattle.io/v1, Kind=Addon controller"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.664589113Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"ccm\", UID:\"e3c9eada-799b-45f4-a9d7-f14df8554a2e\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"295\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/ccm.yaml\""
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.673836546Z" level=info msg="Cluster dns configmap already exists"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.693593 20645 remote_runtime.go:116] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"e22c61e06b2c637e4e6eeb211ac4e303519f5b0efbd63c2172d3f94ef79e036b\": open /run/flannel/subnet.env: no such file or directory"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.693822 20645 kuberuntime_sandbox.go:68] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"e22c61e06b2c637e4e6eeb211ac4e303519f5b0efbd63c2172d3f94ef79e036b\": open /run/flannel/subnet.env: no such file or directory" pod="kube-system/metrics-server-86cbb8457f-dt8lv"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.693912 20645 kuberuntime_manager.go:790] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to setup network for sandbox \"e22c61e06b2c637e4e6eeb211ac4e303519f5b0efbd63c2172d3f94ef79e036b\": open /run/flannel/subnet.env: no such file or directory" pod="kube-system/metrics-server-86cbb8457f-dt8lv"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: E1029 03:01:40.694362 20645 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"metrics-server-86cbb8457f-dt8lv_kube-system(ae4b2f7c-6868-4c65-8dfc-8e9b29b57696)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"metrics-server-86cbb8457f-dt8lv_kube-system(ae4b2f7c-6868-4c65-8dfc-8e9b29b57696)\\\": rpc error: code = Unknown desc = failed to setup network for sandbox \\\"e22c61e06b2c637e4e6eeb211ac4e303519f5b0efbd63c2172d3f94ef79e036b\\\": open /run/flannel/subnet.env: no such file or directory\"" pod="kube-system/metrics-server-86cbb8457f-dt8lv" podUID=ae4b2f7c-6868-4c65-8dfc-8e9b29b57696
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: I1029 03:01:40.846053 20645 network_policy_controller.go:154] Starting network policy controller full sync goroutine
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.978366913Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChartConfig controller"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.978553205Z" level=info msg="Starting helm.cattle.io/v1, Kind=HelmChart controller"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.979498370Z" level=info msg="Starting batch/v1, Kind=Job controller"
Oct 29 03:01:40 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:40.980441128Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"ccm\", UID:\"e3c9eada-799b-45f4-a9d7-f14df8554a2e\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"295\", FieldPath:\"\"}): type: 'Normal' reason: 'AppliedManifest' Applied manifest at \"/var/lib/rancher/k3s/server/manifests/ccm.yaml\""
Oct 29 03:01:41 ubuntu-k3s-master k3s[20645]: I1029 03:01:41.002956 20645 controller.go:611] quota admission added evaluator for: addons.k3s.cattle.io
Oct 29 03:01:41 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:41.076658020Z" level=info msg="Starting /v1, Kind=Node controller"
Oct 29 03:01:41 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:41.076973791Z" level=info msg="Starting /v1, Kind=Service controller"
Oct 29 03:01:41 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:41.077129121Z" level=info msg="Starting /v1, Kind=Pod controller"
Oct 29 03:01:41 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:41.077290859Z" level=info msg="Starting /v1, Kind=Endpoints controller"
Oct 29 03:01:41 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:41.077850495Z" level=info msg="Starting /v1, Kind=Secret controller"
Oct 29 03:01:41 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:41.116546392Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"coredns\", UID:\"8716daf0-1e90-452f-a04a-7ed17c777998\", APIVersion:\"k3s.cattle.io/v1\", ResourceVersion:\"313\", FieldPath:\"\"}): type: 'Normal' reason: 'ApplyingManifest' Applying manifest at \"/var/lib/rancher/k3s/server/manifests/coredns.yaml\""
Oct 29 03:01:41 ubuntu-k3s-master k3s[20645]: I1029 03:01:41.144905 20645 controller.go:611] quota admission added evaluator for: helmcharts.helm.cattle.io
Oct 29 03:01:41 ubuntu-k3s-master k3s[20645]: I1029 03:01:41.349305 20645 kube.go:123] Node controller sync successful
Oct 29 03:01:41 ubuntu-k3s-master k3s[20645]: I1029 03:01:41.349545 20645 vxlan.go:123] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
Oct 29 03:01:41 ubuntu-k3s-master k3s[20645]: time="2021-10-29T03:01:41.358989929Z" level=fatal msg="flannel exited: operation not supported"
Oct 29 03:01:41 ubuntu-k3s-master systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Oct 29 03:01:41 ubuntu-k3s-master systemd[1]: k3s.service: Failed with result 'exit-code'.
Oct 29 03:01:41 ubuntu-k3s-master systemd[1]: k3s.service: Consumed 32.257s CPU time.
Oct 29 03:01:46 ubuntu-k3s-master systemd[1]: k3s.service: Scheduled restart job, restart counter is at 44.
Oct 29 03:01:46 ubuntu-k3s-master systemd[1]: Stopped Lightweight Kubernetes.
Oct 29 03:01:46 ubuntu-k3s-master systemd[1]: k3s.service: Consumed 32.257s CPU time.
Oct 29 03:01:46 ubuntu-k3s-master systemd[1]: Starting Lightweight Kubernetes...
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:47.340823692Z" level=info msg="Starting k3s v1.21.5+k3s2 (724ef700)"
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:47.341547622Z" level=info msg="Cluster bootstrap already complete"
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:47.408536804Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:47.408663708Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:47.409137365Z" level=info msg="Database tables and indexes are up to date"
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:47.427163060Z" level=info msg="Kine listening on unix://kine.sock"
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:47.428346664Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: Flag --insecure-port has been deprecated, This flag has no effect now and will be removed in v1.24.
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: I1029 03:01:47.432841 21077 server.go:656] external host was not specified, using 192.168.86.90
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: I1029 03:01:47.433825 21077 server.go:195] Version: v1.21.5+k3s2
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: I1029 03:01:47.445901 21077 shared_informer.go:240] Waiting for caches to sync for node_authorizer
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: I1029 03:01:47.451686 21077 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: I1029 03:01:47.451809 21077 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: I1029 03:01:47.458354 21077 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: I1029 03:01:47.458454 21077 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: I1029 03:01:47.561397 21077 instance.go:283] Using reconciler: lease
Oct 29 03:01:47 ubuntu-k3s-master k3s[21077]: I1029 03:01:47.700355 21077 rest.go:130] the default service ipfamily for this cluster is: IPv4
Oct 29 03:01:48 ubuntu-k3s-master k3s[21077]: W1029 03:01:48.949755 21077 genericapiserver.go:425] Skipping API node.k8s.io/v1alpha1 because it has no resources.
Oct 29 03:01:48 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:48.954774568Z" level=info msg="Cluster-Http-Server 2021/10/29 03:01:48 http: TLS handshake error from 192.168.86.93:50568: remote error: tls: bad certificate"
Oct 29 03:01:48 ubuntu-k3s-master k3s[21077]: W1029 03:01:48.999493 21077 genericapiserver.go:425] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: W1029 03:01:49.016640 21077 genericapiserver.go:425] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: W1029 03:01:49.045889 21077 genericapiserver.go:425] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: W1029 03:01:49.057907 21077 genericapiserver.go:425] Skipping API flowcontrol.apiserver.k8s.io/v1alpha1 because it has no resources.
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: W1029 03:01:49.086073 21077 genericapiserver.go:425] Skipping API apps/v1beta2 because it has no resources.
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: W1029 03:01:49.086161 21077 genericapiserver.go:425] Skipping API apps/v1beta1 because it has no resources.
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: I1029 03:01:49.126459 21077 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: I1029 03:01:49.126574 21077 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.173440925Z" level=info msg="Waiting for API server to become available"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.173527812Z" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.175431921Z" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.177466489Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --bind-address=127.0.0.1 --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --node-status-update-frequency=1m0s --port=0 --profiling=false"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.181115227Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.181257335Z" level=info msg="To join node to cluster: k3s agent -s https://192.168.86.90:6443 -t ${NODE_TOKEN}"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.184790150Z" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.184933110Z" level=info msg="Run: k3s kubectl"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.288330294Z" level=info msg="Cluster-Http-Server 2021/10/29 03:01:49 http: TLS handshake error from 127.0.0.1:46550: remote error: tls: bad certificate"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.308128082Z" level=info msg="Cluster-Http-Server 2021/10/29 03:01:49 http: TLS handshake error from 127.0.0.1:46556: remote error: tls: bad certificate"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.376057732Z" level=info msg="certificate CN=ubuntu-k3s-master signed by CN=k3s-server-ca@1635475569: notBefore=2021-10-29 02:46:09 +0000 UTC notAfter=2022-10-29 03:01:49 +0000 UTC"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.394614843Z" level=info msg="certificate CN=system:node:ubuntu-k3s-master,O=system:nodes signed by CN=k3s-client-ca@1635475569: notBefore=2021-10-29 02:46:09 +0000 UTC notAfter=2022-10-29 03:01:49 +0000 UTC"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.697029761Z" level=info msg="Module overlay was already loaded"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.697236534Z" level=info msg="Module nf_conntrack was already loaded"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.697301015Z" level=info msg="Module br_netfilter was already loaded"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.697348921Z" level=info msg="Module iptable_nat was already loaded"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: W1029 03:01:49.697460 21077 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.705044630Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Oct 29 03:01:49 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:49.705319421Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:50.709186104Z" level=info msg="Containerd is now running"
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:50.739042794Z" level=info msg="Connecting to proxy" url="wss://127.0.0.1:6443/v1-k3s/connect"
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:50.748936309Z" level=info msg="Handling backend connection request [ubuntu-k3s-master]"
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:50.751170892Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/5b1d238ed00ea9bbae6aa96d867c1264a96ad48b42e55d56570d54d05c8d8a7a/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ubuntu-k3s-master --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:50.752933003Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=ubuntu-k3s-master --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: W1029 03:01:50.755481 21077 server.go:220] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: I1029 03:01:50.806053 21077 server.go:436] "Kubelet version" kubeletVersion="v1.21.5+k3s2"
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: E1029 03:01:50.847821 21077 node.go:161] Failed to retrieve node info: nodes "ubuntu-k3s-master" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: I1029 03:01:50.904364 21077 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
Oct 29 03:01:50 ubuntu-k3s-master k3s[21077]: W1029 03:01:50.904514 21077 manager.go:159] Cannot detect current cgroup on cgroup v2
Oct 29 03:01:51 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:51.502283695Z" level=info msg="Handling backend connection request [ubuntu-k3s-wkr1]"
Oct 29 03:01:51 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:51.504823864Z" level=info msg="Handling backend connection request [ubuntu-k3s-wkr2]"
Oct 29 03:01:52 ubuntu-k3s-master k3s[21077]: E1029 03:01:52.008632 21077 node.go:161] Failed to retrieve node info: nodes "ubuntu-k3s-master" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Oct 29 03:01:53 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:53.981316729Z" level=info msg="Cluster-Http-Server 2021/10/29 03:01:53 http: TLS handshake error from 192.168.86.93:50572: remote error: tls: bad certificate"
Oct 29 03:01:54 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:54.075366683Z" level=error msg="runtime core not ready"
Oct 29 03:01:54 ubuntu-k3s-master k3s[21077]: E1029 03:01:54.075512 21077 node.go:161] Failed to retrieve node info: nodes "ubuntu-k3s-master" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: time="2021-10-29T03:01:55.743474269Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: W1029 03:01:55.928621 21077 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.940412 21077 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.942749 21077 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.944050 21077 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.945336 21077 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.946494 21077 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.947324 21077 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.953110 21077 kubelet.go:404] "Attempting to sync node with API server"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.954794 21077 kubelet.go:272] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.955855 21077 kubelet.go:283] "Adding apiserver pod source"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.958075 21077 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.962150 21077 kuberuntime_manager.go:222] "Container runtime initialized" containerRuntime="containerd" version="v1.4.11-k3s1" apiVersion="v1alpha2"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.964640 21077 server.go:1191] "Started kubelet"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.982576 21077 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.984627 21077 server.go:149] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: E1029 03:01:55.992976 21077 cri_stats_provider.go:369] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.993090 21077 server.go:409] "Adding debug handlers to kubelet server"
Oct 29 03:01:55 ubuntu-k3s-master k3s[21077]: I1029 03:01:55.996010 21077 volume_manager.go:271] "Starting Kubelet Volume Manager"
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: E1029 03:01:55.993109 21077 kubelet.go:1306] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: I1029 03:01:56.013052 21077 desired_state_of_world_populator.go:141] "Desired state populator starts to run"
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: I1029 03:01:56.106950 21077 kuberuntime_manager.go:1044] "Updating runtime config through cri with podcidr" CIDR="10.42.0.0/24"
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: I1029 03:01:56.118048 21077 kubelet_network.go:76] "Updating Pod CIDR" originalPodCIDR="" newPodCIDR="10.42.0.0/24"
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: I1029 03:01:56.122000 21077 kubelet_node_status.go:71] "Attempting to register node" node="ubuntu-k3s-master"
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: I1029 03:01:56.303166 21077 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv4
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: I1029 03:01:56.464550 21077 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv6
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: I1029 03:01:56.464657 21077 status_manager.go:157] "Starting to sync pod status with apiserver"
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: I1029 03:01:56.464722 21077 kubelet.go:1846] "Starting kubelet main sync loop"
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: E1029 03:01:56.464909 21077 kubelet.go:1870] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: E1029 03:01:56.570995 21077 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: E1029 03:01:56.775965 21077 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 29 03:01:56 ubuntu-k3s-master k3s[21077]: I1029 03:01:56.960547 21077 apiserver.go:52] "Watching apiserver"
Oct 29 03:01:57 ubuntu-k3s-master k3s[21077]: E1029 03:01:57.176935 21077 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 29 02:42:48 ubuntu-k3s-wkr1 systemd[1]: k3s-node.service: Main process exited, code=exited, status=1/FAILURE
Oct 29 02:42:48 ubuntu-k3s-wkr1 systemd[1]: k3s-node.service: Failed with result 'exit-code'.
Oct 29 02:42:48 ubuntu-k3s-wkr1 systemd[1]: Stopped Lightweight Kubernetes.
Oct 29 02:42:48 ubuntu-k3s-wkr1 systemd[1]: k3s-node.service: Consumed 2.549s CPU time.
-- Boot 916f2a3c80474e0794dd8ba58cda8c99 --
Oct 29 02:47:13 ubuntu-k3s-wkr1 systemd[1]: Starting Lightweight Kubernetes...
Oct 29 02:47:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:47:14Z" level=info msg="Acquiring lock file /var/lib/rancher/k3s/data/.lock"
Oct 29 02:47:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:47:14Z" level=info msg="Preparing data dir /var/lib/rancher/k3s/data/5b1d238ed00ea9bbae6aa96d867c1264a96ad48b42e55d56570d54d05c8d8a7a"
Oct 29 02:47:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:47:22.175862974Z" level=info msg="Starting k3s agent v1.21.5+k3s2 (724ef700)"
Oct 29 02:47:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:47:22.178068129Z" level=info msg="Running load balancer 127.0.0.1:6444 -> [192.168.86.90:6443]"
Oct 29 02:47:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:47:22.526144774Z" level=error msg="Failed to configure agent: https://127.0.0.1:6444/v1-k3s/serving-kubelet.crt: 503 Service Unavailable"
Oct 29 02:47:27 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:47:27.533919075Z" level=error msg="Failed to configure agent: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
Oct 29 02:47:32 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:47:32.622221332Z" level=error msg="Failed to configure agent: https://127.0.0.1:6444/v1-k3s/serving-kubelet.crt: 503 Service Unavailable"
Oct 29 02:47:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:47:37.752032108Z" level=error msg="Failed to configure agent: https://127.0.0.1:6444/v1-k3s/serving-kubelet.crt: 503 Service Unavailable"
Oct 29 02:47:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:47:42.909486249Z" level=error msg="Failed to configure agent: https://127.0.0.1:6444/v1-k3s/serving-kubelet.crt: 503 Service Unavailable"
Oct 29 02:47:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:47:47.916449232Z" level=error msg="Failed to configure agent: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
Oct 29 02:47:52 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:47:52.996412360Z" level=error msg="Failed to configure agent: https://127.0.0.1:6444/v1-k3s/serving-kubelet.crt: 503 Service Unavailable"
Oct 29 02:47:58 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:47:58.073757931Z" level=error msg="Failed to configure agent: https://127.0.0.1:6444/v1-k3s/serving-kubelet.crt: 503 Service Unavailable"
Oct 29 02:48:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:04.514121234Z" level=info msg="Module overlay was already loaded"
Oct 29 02:48:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:04.552447652Z" level=info msg="Module br_netfilter was already loaded"
Oct 29 02:48:04 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:04.571599 3222 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Oct 29 02:48:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:04.574636343Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
Oct 29 02:48:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:04.574948304Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
Oct 29 02:48:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:04.575553673Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 131072"
Oct 29 02:48:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:04.579794715Z" level=info msg="Logging containerd to /var/lib/rancher/k3s/agent/containerd/containerd.log"
Oct 29 02:48:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:04.580150381Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Oct 29 02:48:05 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:05.588657505Z" level=info msg="Containerd is now running"
Oct 29 02:48:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:10.679898951Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:48:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:10.693189430Z" level=error msg="Failed to connect to proxy" error="websocket: bad handshake"
Oct 29 02:48:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:10.693286856Z" level=error msg="Remotedialer proxy error" error="websocket: bad handshake"
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:15.684241765Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:15.694078034Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:15.709869692Z" level=info msg="Running kubelet --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --cni-bin-dir=/var/lib/rancher/k3s/data/5b1d238ed00ea9bbae6aa96d867c1264a96ad48b42e55d56570d54d05c8d8a7a/bin --cni-conf-dir=/var/lib/rancher/k3s/agent/etc/cni/net.d --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --container-runtime=remote --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ubuntu-k3s-wkr1 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --node-labels= --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key"
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:15.712385722Z" level=info msg="Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=ubuntu-k3s-wkr1 --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables"
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: Flag --cloud-provider has been deprecated, will be removed in 1.23, in favor of removing cloud provider code from Kubelet.
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: Flag --cni-bin-dir has been deprecated, will be removed along with dockershim.
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: Flag --cni-conf-dir has been deprecated, will be removed along with dockershim.
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:15.713560 3222 server.go:220] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:15.761400 3222 server.go:436] "Kubelet version" kubeletVersion="v1.21.5+k3s2"
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:15.852608 3222 node.go:161] Failed to retrieve node info: nodes "ubuntu-k3s-wkr1" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:15.857259 3222 dynamic_cafile_content.go:167] Starting client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt
Oct 29 02:48:15 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:15.857391 3222 manager.go:159] Cannot detect current cgroup on cgroup v2
Oct 29 02:48:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:16.937270 3222 node.go:161] Failed to retrieve node info: nodes "ubuntu-k3s-wkr1" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Oct 29 02:48:19 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:19.123342 3222 node.go:161] Failed to retrieve node info: nodes "ubuntu-k3s-wkr1" is forbidden: User "system:kube-proxy" cannot get resource "nodes" in API group "" at the cluster scope
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:20.883132 3222 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.892056 3222 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.893109 3222 container_manager_linux.go:291] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.893333 3222 container_manager_linux.go:296] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:remote CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none Rootless:false}
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.893599 3222 topology_manager.go:120] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.893649 3222 container_manager_linux.go:327] "Initializing Topology Manager" policy="none" scope="container"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.893681 3222 container_manager_linux.go:332] "Creating device plugin manager" devicePluginEnabled=true
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.894832 3222 kubelet.go:404] "Attempting to sync node with API server"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.894929 3222 kubelet.go:272] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.895082 3222 kubelet.go:283] "Adding apiserver pod source"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.895150 3222 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.898048 3222 kuberuntime_manager.go:222] "Container runtime initialized" containerRuntime="containerd" version="v1.4.11-k3s1" apiVersion="v1alpha2"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:20.898737 3222 probe.go:268] Flexvolume plugin directory at /usr/libexec/kubernetes/kubelet-plugins/volume/exec/ does not exist. Recreating.
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.900758 3222 server.go:1191] "Started kubelet"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.905846 3222 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.912958 3222 volume_manager.go:271] "Starting Kubelet Volume Manager"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.915215 3222 desired_state_of_world_populator.go:141] "Desired state populator starts to run"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.917322 3222 server.go:149] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:20.918029 3222 cri_stats_provider.go:369] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:20.918891 3222 kubelet.go:1306] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:20.919806 3222 server.go:409] "Adding debug handlers to kubelet server"
Oct 29 02:48:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:20.999050 3222 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ubuntu-k3s-wkr1\" not found" node="ubuntu-k3s-wkr1"
Oct 29 02:48:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:21.014933 3222 kubelet.go:2291] "Error getting node" err="node \"ubuntu-k3s-wkr1\" not found"
Oct 29 02:48:21 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:21.023729 3222 kubelet_node_status.go:71] "Attempting to register node" node="ubuntu-k3s-wkr1"
Oct 29 02:48:21 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:21.039481 3222 kubelet_node_status.go:74] "Successfully registered node" node="ubuntu-k3s-wkr1"
Oct 29 02:48:21 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:21.246005 3222 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv4
Oct 29 02:48:21 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:21.412186 3222 kubelet_network_linux.go:56] "Initialized protocol iptables rules." protocol=IPv6
Oct 29 02:48:21 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:21.412295 3222 status_manager.go:157] "Starting to sync pod status with apiserver"
Oct 29 02:48:21 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:21.412355 3222 kubelet.go:1846] "Starting kubelet main sync loop"
Oct 29 02:48:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:21.412534 3222 kubelet.go:1870] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 02:48:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:21.513293 3222 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 29 02:48:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:21.713598 3222 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 29 02:48:21 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:21.896439 3222 apiserver.go:52] "Watching apiserver"
Oct 29 02:48:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:22.113764 3222 kubelet.go:1870] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 29 02:48:22 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:22.655415 3222 cpu_manager.go:199] "Starting CPU manager" policy="none"
Oct 29 02:48:22 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:22.655483 3222 cpu_manager.go:200] "Reconciling" reconcilePeriod="10s"
Oct 29 02:48:22 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:22.655550 3222 state_mem.go:36] "Initialized new in-memory state store"
Oct 29 02:48:22 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:22.668452 3222 policy_none.go:44] "None policy: Start"
Oct 29 02:48:22 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:22.702881 3222 manager.go:600] "Failed to retrieve checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
Oct 29 02:48:22 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:22.732962 3222 plugin_manager.go:114] "Starting Kubelet Plugin Manager"
Oct 29 02:48:22 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:22.928543 3222 reconciler.go:157] "Reconciler: start to sync state"
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:23.705211 3222 node.go:172] Successfully retrieved node IP: 192.168.86.91
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:23.705391 3222 server_others.go:141] Detected node IP 192.168.86.91
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:23.726742 3222 server_others.go:207] kube-proxy running in dual-stack mode, IPv4-primary
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:23.726844 3222 server_others.go:213] Using iptables Proxier.
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:23.726894 3222 server_others.go:220] creating dualStackProxier for iptables.
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:23.726942 3222 server_others.go:519] detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:23.728410 3222 server.go:643] Version: v1.21.5+k3s2
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:23.729807 3222 config.go:315] Starting service config controller
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:23.729891 3222 shared_informer.go:240] Waiting for caches to sync for service config
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:23.729967 3222 config.go:224] Starting endpoint slice config controller
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:23.729999 3222 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:23.759570 3222 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:23.765115 3222 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:23.830936 3222 shared_informer.go:247] Caches are synced for endpoint slice config
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:23.830958 3222 shared_informer.go:247] Caches are synced for service config
Oct 29 02:48:23 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:23.846318673Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:48:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:24.895828101Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:48:24 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:24.910105 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:48:24 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:24.910171 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:48:24 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:24.910199 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:48:24 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:24.910474 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:48:24 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:24.910755 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:48:24 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:24.910756 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:48:24 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:24.910903 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1beta1.EndpointSlice ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:48:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:29.896606709Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:48:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:29.897584429Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:48:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:29.897726465Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:31.006778 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: the server is currently unable to handle the request (get nodes)
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:31.040794 3222 reflector.go:138] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Failed to watch *v1.Pod: failed to list *v1.Pod: the server is currently unable to handle the request (get pods)
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:31.111400 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:31.120934 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:31.135430 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:31.152527 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:31.155126 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:31.163849 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:31.164554 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:31.272872 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: the server is currently unable to handle the request (get endpointslices.discovery.k8s.io)
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:31.547262 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: the server is currently unable to handle the request (get runtimeclasses.node.k8s.io)
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:31.772946 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:48:31 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:31.885127939Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:48:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:32.136205 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:48:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:32.181737 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: the server is currently unable to handle the request (get csidrivers.storage.k8s.io)
Oct 29 02:48:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:32.338823 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:48:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:32.807273 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:kube-proxy" cannot list resource "services" in API group "" at the cluster scope
Oct 29 02:48:33 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:33.897598817Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:48:34 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:34.026894 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
Oct 29 02:48:34 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:34.898078301Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:48:34 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:34.926274961Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:48:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:35.910756270Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:48:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:37.924926946Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:48:38 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:38.143792 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:kube-proxy" cannot list resource "services" in API group "" at the cluster scope
Oct 29 02:48:38 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:38.602913 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
Oct 29 02:48:39 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:39.930101321Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:48:39 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:39.939326813Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:48:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:41.982536852Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:48:43 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:43.995104644Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:48:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:45.529897832Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:48:45 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:45.536564 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:48:45 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:45.537796 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:48:45 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:45.538299 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:48:45 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:45.539760 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:48:45 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:48:45.540159 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:48:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:50.534282865Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:48:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:50.537034059Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:48:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:50.537199778Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:48:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:51.313032 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:48:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:51.316359 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:48:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:51.319381 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:48:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:51.322689 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:48:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:51.325167 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:48:51 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:48:51.325278 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:48:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:51.761706 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: the server is currently unable to handle the request (get runtimeclasses.node.k8s.io)
Oct 29 02:48:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:51.809733 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:48:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:51.908154 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:48:52 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:52.035668573Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:48:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:52.151535 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: the server is currently unable to handle the request (get endpointslices.discovery.k8s.io)
Oct 29 02:48:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:52.316715 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:48:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:52.320030 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:48:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:52.335929 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:48:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:52.365300 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:48:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:52.385885 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:48:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:52.385996 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:48:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:52.461084 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: the server is currently unable to handle the request (get nodes)
Oct 29 02:48:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:52.578817 3222 reflector.go:138] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Failed to watch *v1.Pod: failed to list *v1.Pod: the server is currently unable to handle the request (get pods)
Oct 29 02:48:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:48:52.581102 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: the server is currently unable to handle the request (get csidrivers.storage.k8s.io)
Oct 29 02:48:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:54.051177204Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:48:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:55.538066625Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:48:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:55.550889360Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:48:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:56.062465792Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:48:58 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:48:58.073312603Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:00.093813459Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:00.555892216Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:49:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:02.109785337Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:03 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:03.372190 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Oct 29 02:49:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:04.120187620Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:06.131859286Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:07 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:49:07.687204 3222 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
Oct 29 02:49:07 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:49:07.844242 3222 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
Oct 29 02:49:07 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:49:07.980147 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1beta1.EndpointSlice ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:49:08 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:08.019909672Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:49:08 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:49:08.028285 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:49:08 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:49:08.029768 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:49:08 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:49:08.029768 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:49:08 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:49:08.029930 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:49:08 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:49:08.029971 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:49:08 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:49:08.031981 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:49:13 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:13.020665605Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:49:13 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:13.021783641Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:49:13 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:13.021956971Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:49:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:13.669475 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:13.673484 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:13.676943 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:13.679205 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:13.681510 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:13 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:49:13.681607 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:49:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:14.173722240Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:14.430801 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:14.433228 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:14.446392 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:14.458560 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:14.470004 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:14.470684 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:49:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:14.710352 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:49:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:14.913607 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:49:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:15.316333 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:49:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:16.186657498Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:18.022530848Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:49:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:18.041508693Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:49:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:18.207805857Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:20.222353993Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:22.236339479Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:23 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:23.046178919Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:49:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:24.249344697Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:26 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:26.261373396Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:28.274600556Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:30 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:49:30.404584 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:49:30 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:49:30.405046 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:49:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:30.444900833Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:49:30 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:49:30.451043 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:49:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:35.445925880Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:49:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:35.447092808Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:49:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:35.447262732Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.275562 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.278927 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.281911 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.284724 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.287266 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:49:36.287360 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:36.497849378Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.529833 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.531823 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.543637 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.554933 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.569583 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.569677 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.731378 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: the server is currently unable to handle the request (get csidrivers.storage.k8s.io)
Oct 29 02:49:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:36.735405 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: the server is currently unable to handle the request (get nodes)
Oct 29 02:49:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:37.307237 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:49:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:37.509730 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:49:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:37.912480 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:49:38 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:38.523540109Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:40.448214861Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:49:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:40.465239724Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:49:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:40.537764097Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:42.552214956Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:44.083509 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:kube-proxy" cannot list resource "services" in API group "" at the cluster scope
Oct 29 02:49:44 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:44.565724491Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:45.469193146Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:49:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:46.658077944Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:48 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:48.669202468Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:49:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:51.016276278Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:49:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:56.018480656Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:49:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:56.019220684Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:49:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:56.019338794Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:49:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:56.877738 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:56.881714 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:56.885775 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:56.887997 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:56.890328 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:49:56 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:49:56.890434 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:49:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:57.625904 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:57.627946 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:57.645534 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:57.659045 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:57.678046 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:57.678728 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:49:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:57.782196 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:49:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:57.915148 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:49:58 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:58.021590 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: the server is currently unable to handle the request (get endpointslices.discovery.k8s.io)
Oct 29 02:49:58 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:49:58.068653420Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:49:58 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:58.120464 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:49:58 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:49:58.523228 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:00.084273064Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:01 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:01.020282946Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:50:01 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:01.040632645Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:50:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:02.095499111Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:04.107119789Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:06.045798853Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:50:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:06.120274686Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:08 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:08.138927397Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:10.157780248Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:11 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:11.628414099Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:50:11 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:50:11.638242 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:50:11 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:50:11.638242 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:50:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:16.629208532Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:50:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:16.630339817Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:50:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:16.630512593Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:17.737352 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:17.747303 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:17.762276 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:17.776680 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:17.788179 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:17.788275 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:17.890548 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:17.900539 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:17.920264 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:17.934740 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:17.946292 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:50:17.946754 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:50:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:17.957774 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:18.166454 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:18.214417972Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:18.569901 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:18.760735 3222 reflector.go:138] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Failed to watch *v1.Pod: failed to list *v1.Pod: the server is currently unable to handle the request (get pods)
Oct 29 02:50:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:20.253096670Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:21 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:21.631707583Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:50:21 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:21.649245786Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:50:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:22.264707600Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:24.275705663Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:26 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:26.288842379Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:26 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:26.654158091Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:50:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:28.303119310Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:29 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:50:29.478676 3222 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
Oct 29 02:50:29 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:50:29.482661 3222 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
Oct 29 02:50:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:30.317530193Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:32 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:32.159411641Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:50:32 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:50:32.169360 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:50:32 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:50:32.169620 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1beta1.EndpointSlice ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:50:32 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:50:32.169620 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:50:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:37.160218536Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:50:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:37.161378026Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:50:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:37.161530284Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:50:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:37.775094 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:50:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:37.778487 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:50:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:37.781570 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:50:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:37.784743 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:50:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:37.787818 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:50:37 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:50:37.787921 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:50:38 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:38.314710 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:50:38 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:38.368215841Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:38 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:38.812802 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:38 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:38.831674 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:38 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:38.837027 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:38 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:38.861256 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:38 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:38.879085 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:38 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:38.898090 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:38 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:38.898224 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:50:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:39.026746 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:39.428798 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:40.380542770Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:42.161945158Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:50:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:42.181572847Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:50:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:42.392877481Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:44 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:44.405773388Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:46.418593285Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:47.186721876Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:50:48 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:48.464898529Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:50.483614160Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:52 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:52.594650392Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:50:53 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:53.104547801Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:50:53 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:50:53.117379 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:50:53 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:50:53.117379 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:50:58 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:58.104843933Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:50:58 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:58.105831666Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:50:58 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:58.106000850Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:50:58 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:58.877826 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:50:58 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:58.881246 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:50:58 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:58.884371 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:50:58 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:58.887478 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:50:58 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:58.890174 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:50:58 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:50:58.890275 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:50:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:50:59.633977048Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:59.917195 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:50:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:59.945130 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:59.951360 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:59.971154 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:59.983847 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:59.996520 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:50:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:50:59.996618 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:51:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:00.132245 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:51:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:00.534993 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:51:01 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:01.647399006Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:03 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:03.106978222Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:51:03 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:03.124880275Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:51:03 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:03.659011792Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:04.823659 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
Oct 29 02:51:05 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:05.669850411Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:07.680530541Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:08 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:08.130318656Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:51:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:09.692188666Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:11 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:11.704331607Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:13 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:13.849484971Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:14.716441750Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:51:14 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:51:14.723651 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:51:19 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:19.716727268Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:51:19 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:19.717720150Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:51:19 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:19.717888520Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:51:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:20.163293 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:51:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:20.166133 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:51:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:20.168954 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:51:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:20.171706 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:51:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:20.174560 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:51:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:51:20.174681 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:51:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:20.892713914Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:51:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:21.048468 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:51:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:21.050751 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:51:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:21.060978 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:51:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:21.072477 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:51:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:21.082893 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:51:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:21.082977 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:51:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:21.177060 3222 reflector.go:138] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Failed to watch *v1.Pod: failed to list *v1.Pod: the server is currently unable to handle the request (get pods)
Oct 29 02:51:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:21.189911 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:51:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:21.399722 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:51:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:21.802916 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:51:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:22.905131117Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:24.718308909Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:51:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:24.736000567Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:51:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:24.920418556Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:26 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:26.931743513Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:28.942052928Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:29.741395118Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:51:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:30.956382061Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:32.626454 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Oct 29 02:51:32 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:32.974880260Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:34 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:34.985646267Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:37.120254706Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:51:37 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:51:37.130852 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:51:37 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:51:37.133908 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:51:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:42.120631202Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:51:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:42.123170741Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:51:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:42.123319962Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:51:43 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:43.061813 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:51:43 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:43.065845 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:51:43 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:43.069629 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:51:43 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:43.072054 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:51:43 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:43.074547 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:51:43 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:51:43.074653 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:51:43 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:43.173302295Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:51:43 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:43.796288 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: the server is currently unable to handle the request (get nodes)
Oct 29 02:51:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:44.092509 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:51:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:44.142464 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:51:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:44.144727 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:51:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:44.155855 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:51:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:44.167854 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:51:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:44.178701 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:51:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:44.178802 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:51:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:44.306050 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:51:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:51:44.709327 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:51:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:45.185295656Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:47.123725849Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:51:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:47.145989154Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:51:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:47.198747340Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:49 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:49.212671723Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:51.230707410Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:52 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:52.150929201Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:51:53 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:53.297490586Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:55.308136381Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:57 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:57.393075820Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:51:57 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:51:57.807556687Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:51:57 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:51:57.818766 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:52:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:02.808064272Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:52:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:02.809155950Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:52:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:02.809310209Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:52:03 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:03.414656 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:03 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:03.418888 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:03 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:03.421364 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:03 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:03.423713 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:03 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:03.426078 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:03 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:52:03.426190 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:52:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:04.259699 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:04.261466 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: the server is currently unable to handle the request (get endpointslices.discovery.k8s.io)
Oct 29 02:52:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:04.261696 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:04.272265 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:04.282842 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:04.293589 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:04.293679 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:52:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:04.431892386Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:04.442644 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:52:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:04.661325 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:52:05 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:05.063988 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:52:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:06.445148159Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:07.810406732Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:52:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:07.827717517Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:52:08 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:08.462226203Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:10.474017035Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:12.492721283Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:12.831927846Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:52:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:14.504302381Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:16.516831742Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:18.404414418Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:52:18 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:52:18.411636 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:52:23 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:23.404979317Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:52:23 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:23.406075459Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:52:23 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:23.406238236Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:52:24 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:24.113001 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:24 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:24.116956 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:24 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:24.120394 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:24 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:24.122763 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:24 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:24.125158 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:24 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:52:24.125271 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:52:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:24.568647601Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:24 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:24.685117 3222 reflector.go:138] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Failed to watch *v1.Pod: failed to list *v1.Pod: the server is currently unable to handle the request (get pods)
Oct 29 02:52:24 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:24.698144 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:52:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:25.194247 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:52:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:25.335923 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:25.339297 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:25.352349 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:25.375735 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:25.390872 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:25.391729 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:52:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:25.406809 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:52:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:25.809329 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:52:26 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:26.593682401Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:28.407376550Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:52:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:28.425897814Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:52:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:28.609169166Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:30.621917243Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:32 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:32.634499968Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:33 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:33.429625588Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:52:34 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:34.673446153Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:36 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:36.693024038Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:38 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:38.770189834Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:39 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:39.106277817Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:52:39 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:52:39.115486 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:52:39 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:52:39.115486 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:52:39 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:52:39.115591 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:52:44 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:44.107311177Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:52:44 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:44.108476170Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:52:44 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:44.108640855Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:52:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:44.529218 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:44.533217 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:44.535733 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:44.538271 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:44.540640 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:52:44 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:52:44.540747 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:52:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:45.438373 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:45.440302 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:45.449989 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:45.460515 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:45.470843 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:45.470918 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:52:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:45.559633 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:52:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:45.762562 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:52:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:45.815688513Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:52:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:46.165590 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:52:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:47.828325313Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:49 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:49.109616953Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:52:49 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:49.132397362Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:52:49 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:49.839478038Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:51.851106378Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:53 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:52:53.811201 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
Oct 29 02:52:53 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:53.861147893Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:54.137119706Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:52:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:55.873821212Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:57 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:57.910082435Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:52:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:52:59.568714405Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:53:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:04.569481200Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:53:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:04.570573895Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:53:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:04.570731595Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:53:05 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:05.135796 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:53:05 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:05.139828 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:53:05 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:05.142651 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:53:05 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:05.145256 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:53:05 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:05.147957 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:53:05 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:53:05.148072 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:53:05 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:05.952429380Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:06.132485 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: the server is currently unable to handle the request (get csidrivers.storage.k8s.io)
Oct 29 02:53:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:06.166934 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:06.376360 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:06.515149 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:06.518023 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:06.538830 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:06.553523 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:06.568371 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:06.569118 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:53:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:06.640145 3222 reflector.go:138] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Failed to watch *v1.Pod: failed to list *v1.Pod: the server is currently unable to handle the request (get pods)
Oct 29 02:53:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:06.789419 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:07.964336191Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:09.571901678Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:53:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:09.589744131Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:53:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:09.974210837Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:11 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:11.983946209Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:13 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:13.994360941Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:14.594619071Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:53:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:16.022670121Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:18.032635616Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:20.067877748Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:20.235778338Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:53:20 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:53:20.245090 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:53:20 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:53:20.245091 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:53:22 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:53:22.666998 3222 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Oct 29 02:53:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:25.237044651Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:53:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:25.238077831Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:53:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:25.238234680Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.187576 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.190055 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.203562 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.218792 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.244355 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:53:26.244731 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.275917 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.488124 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.576746 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.620590 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.622450 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.633114 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.643991 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.654657 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.654750 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:53:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:26.908484 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:27 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:27.106414890Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:27 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:27.385311 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: the server is currently unable to handle the request (get runtimeclasses.node.k8s.io)
Oct 29 02:53:27 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:27.711932 3222 controller.go:144] failed to ensure lease exists, will retry in 1.6s, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:29.125585973Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:30.239133448Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:53:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:30.257429936Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:53:31 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:31.136197779Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:33 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:33.147282098Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:35.159634332Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:35.261504546Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:53:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:37.171893544Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:39 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:39.183420744Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:41.015709063Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:53:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:46.016411050Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:53:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:46.017602581Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:53:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:46.017763856Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:53:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:46.872809 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:53:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:46.876295 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:53:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:46.879839 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:53:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:46.882742 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:53:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:46.884982 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:53:46 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:53:46.885075 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:53:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:47.237778 3222 reflector.go:138] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Failed to watch *v1.Pod: failed to list *v1.Pod: the server is currently unable to handle the request (get pods)
Oct 29 02:53:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:47.244657707Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:47.715630 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:47.719116 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:47.735890 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:47.759028 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:47.782920 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:53:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:47.783040 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:53:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:47.788110 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: the server is currently unable to handle the request (get endpointslices.discovery.k8s.io)
Oct 29 02:53:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:47.909938 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:48 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:48.116521 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:48 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:53:48.519066 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:53:49 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:49.273205679Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:51.018687985Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:53:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:51.036506115Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:53:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:51.284684718Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:53 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:53.299670115Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:55.311813243Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:56.041456292Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:53:57 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:57.337555078Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:53:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:53:59.350265439Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:01 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:01.487645615Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:01 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:01.723857542Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:54:01 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:54:01.733134 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:54:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:06.724820129Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:54:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:06.725990033Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:54:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:06.726160789Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:07.362724 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:07.366172 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:07.369100 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:07.371968 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:07.374763 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:54:07.374880 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:07.800963 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:07.842296 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:07.844833 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:07.859204 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:07.878528 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:07.892787 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:07.892883 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:54:08 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:08.086858 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: the server is currently unable to handle the request (get csidrivers.storage.k8s.io)
Oct 29 02:54:08 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:08.436892 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:54:08 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:08.530123041Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:08 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:08.646346 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:54:09 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:09.049185 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:54:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:10.562661768Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:11 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:11.726335860Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:54:11 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:11.745260817Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:54:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:12.574313799Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:14.584995434Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:16.596628595Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:16.750272063Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:54:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:18.613379203Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:20.625205341Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:22.703537382Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:54:22 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:54:22.712259 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:54:22 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:54:22.712357 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:54:27 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:27.704415400Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:54:27 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:27.705709323Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:54:27 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:27.705874876Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:28.498252 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:28.501301 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:28.504380 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:28.507363 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:28.509995 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:54:28.510091 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:28.767951480Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:28.956067 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:28.958639 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:28.970008 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:28.981554 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:28.994548 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:28 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:28.995289 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:54:29 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:29.515119 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: the server is currently unable to handle the request (get endpointslices.discovery.k8s.io)
Oct 29 02:54:29 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:29.528582 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:54:29 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:29.731929 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:54:30 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:30.135440 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:54:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:30.781331017Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:32 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:32.706983887Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:54:32 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:32.725156829Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:54:32 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:32.793492061Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:34 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:34.806369372Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:36 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:36.819877722Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:37.729625234Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:54:38 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:38.866761960Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:40.876992333Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:42.923142473Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:43 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:43.555562837Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:54:43 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:54:43.563785 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:54:43 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:54:43.565164 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:54:48 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:48.556369736Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:54:48 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:48.557480274Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:54:48 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:48.557639568Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:54:48 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:48.841486 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:48 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:48.845221 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:48 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:48.848124 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:48 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:48.850712 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:48 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:48.853250 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:54:48 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:54:48.853363 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:54:49 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:49.895688 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:54:49 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:49.967925378Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:50 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:50.045379 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:50 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:50.047243 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:50 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:50.058209 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:50 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:50.068955 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:50 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:50.079551 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:54:50 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:50.079651 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:54:50 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:50.110246 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:54:50 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:50.512968 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:54:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:54:51.315190 3222 controller.go:144] failed to ensure lease exists, will retry in 1.6s, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:54:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:51.979749302Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:53 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:53.557788181Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:54:53 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:53.573333799Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:54:53 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:53.990647201Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:56.000744461Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:58 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:58.011419685Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:54:58 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:54:58.578980987Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:55:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:00.038934095Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:02.049976917Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:04.235899138Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:55:04 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:55:04.244238 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:55:04 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:55:04.244433 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:55:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:09.237210989Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:55:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:09.238289937Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:55:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:09.238537767Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:55:10 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:10.192893 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:10 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:10.198175 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:10 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:10.215697 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:10 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:10.232679 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:10 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:10.245225 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:10 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:55:10.245587 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:55:10 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:10.265619 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:10.284755153Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:10 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:10.468476 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:10 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:10.518594 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: the server is currently unable to handle the request (get runtimeclasses.node.k8s.io)
Oct 29 02:55:10 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:10.872712 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:11 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:11.134085 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:11 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:11.148942 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:11 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:11.164632 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:11 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:11.175857 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:11 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:11.187240 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:11 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:11.187334 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:55:11 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:11.688670 3222 controller.go:144] failed to ensure lease exists, will retry in 1.6s, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:12.310808225Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:13.554200 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
Oct 29 02:55:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:14.239675118Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:55:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:14.259575841Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:55:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:14.323860427Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:16.337623081Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:18.351372026Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:19 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:19.264786573Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:55:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:20.369637289Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:22.379844501Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:24.438169497Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:24 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:55:24.735478 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:55:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:24.783831479Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:55:24 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:55:24.793350 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:55:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:29.784302515Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:55:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:29.785362964Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:55:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:29.785517259Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:55:30 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:30.401193 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:55:30 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:30.405293 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:55:30 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:30.409218 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:55:30 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:30.411436 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:55:30 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:30.413805 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:55:30 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:55:30.413902 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:55:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:31.242786 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:31.244910 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:31.256292 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:31.267648 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:31.278426 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:31.278531 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:55:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:31.342118 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:55:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:31.434501 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:31 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:31.471736586Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:31.653821 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:32.056839 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:33 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:33.482764091Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:34 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:34.785678495Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:55:34 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:34.802764883Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:55:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:35.494230819Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:37.505549663Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:39 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:39.521517134Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:39 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:39.807377810Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:55:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:41.543100422Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:43 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:43.554511410Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:44 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:55:44.347218 3222 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
Oct 29 02:55:44 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:55:44.357362 3222 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
Oct 29 02:55:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:45.436298908Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:55:45 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:55:45.442266 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:55:45 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:55:45.443505 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1beta1.EndpointSlice ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:55:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:50.436871966Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:55:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:50.437940157Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:55:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:50.438107266Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:55:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:51.430173 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:51.432368 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:51.442523 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:51.453295 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:51.464270 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:51 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:55:51.464603 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:55:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:51.476494 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:51.598840340Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:51 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:51.691689 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:52.101786 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:52.334204 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:52.343731 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:52.358683 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:52.374812 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:52.388773 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:55:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:52.389442 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:55:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:55:52.917124 3222 controller.go:144] failed to ensure lease exists, will retry in 1.6s, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:55:53 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:53.612642258Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:55.438519559Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:55:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:55.455076191Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:55:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:55.626428362Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:57 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:57.638688371Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:55:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:55:59.653300864Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:00.459681724Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:56:01 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:01.700568923Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:03 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:03.710607143Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:05 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:05.833699827Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:06.203592807Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:56:06 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:56:06.213191 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:56:06 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:56:06.213320 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:56:06 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:56:06.213191 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:56:11 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:11.204244597Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:56:11 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:11.205425213Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:56:11 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:11.205607359Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:56:11 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:11.763132 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:56:11 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:11.766544 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:56:11 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:11.769779 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:56:11 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:11.772853 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:56:11 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:11.775569 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:56:11 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:56:11.775672 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:56:12 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:12.433651 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:12 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:12.439380 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:12 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:12.452046 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:12 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:12.463868 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:12 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:12.475150 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:12 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:12.475293 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:56:12 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:12.824983 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:12.869687419Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:13.028419 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:13.430814 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:14.883667731Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:16.205985984Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:56:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:16.225784312Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:56:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:16.894537482Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:18.906697453Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:20.916706237Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:21 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:21.230801469Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:56:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:22.965574237Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:24.984289235Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:26 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:26.711456576Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:56:26 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:56:26.718935 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:56:31 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:31.711724093Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:56:31 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:31.712921414Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:56:31 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:31.713241540Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:56:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:32.271712 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:56:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:32.275649 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:56:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:32.278243 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:56:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:32.280755 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:56:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:32.283347 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:56:32 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:56:32.283454 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:56:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:32.678202 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: the server is currently unable to handle the request (get nodes)
Oct 29 02:56:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:32.919529 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:56:33 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:33.024603348Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:33 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:33.300440 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:33 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:33.509812 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:33 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:33.521292 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:33 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:33.529226 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:33 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:33.552561 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:33 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:33.568991 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:33 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:33.582246 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:33 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:33.582337 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:56:33 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:33.924817 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:35.035970471Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:36 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:36.714141764Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:56:36 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:36.732580526Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:56:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:37.045170668Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:39 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:39.056488447Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:41.067543666Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:41 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:41.631177 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
Oct 29 02:56:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:41.737609433Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:56:43 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:43.078774762Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:44.737272 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Oct 29 02:56:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:45.105817660Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:47.115406455Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:49 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:49.123653584Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:56:49 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:56:49.135710 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:56:49 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:56:49.135889 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:56:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:54.124335990Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:56:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:54.125440405Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:56:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:54.125761494Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:55.174584020Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.563318 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.564949 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.575718 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.586697 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.597970 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:56:55.598565 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.609276 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.626369 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.628856 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.640795 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.651722 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.664074 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.664209 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:56:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:55.900049 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:56:56.303414 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:56:57 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:57.187446437Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:56:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:59.126252490Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:56:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:59.145183367Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:56:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:56:59.201522468Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:01 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:01.216121675Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:03 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:03.228520381Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:04.149514132Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:57:05 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:05.250824738Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:07.262043551Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:09.375247060Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:09 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:57:09.712709 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:57:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:09.748554284Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:57:09 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:57:09.763573 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:57:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:14.749175472Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:57:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:14.750207536Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:57:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:14.750379145Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:57:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:15.466730 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:15.468802 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:15.470711 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:15.472737 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:15.475174 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:15 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:57:15.475266 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:57:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:15.708193 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:15.711262 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:15.721946 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:15.735859 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:15.763627 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:15.763763 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:57:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:16.444213707Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:16.505176 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:57:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:16.708530 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:57:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:17.111576 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:57:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:17.223412 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:57:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:18.469547159Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:19 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:19.750646673Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:57:19 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:19.770327254Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:57:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:20.480931345Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:22.492602936Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:24.503098186Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:24.774905353Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:57:26 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:26.521840558Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:26 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:57:26.811332 3222 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
Oct 29 02:57:26 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:57:26.816021 3222 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
Oct 29 02:57:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:28.542701943Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:30.180135610Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:57:30 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:57:30.188248 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1beta1.EndpointSlice ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:57:30 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:57:30.188519 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:57:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:35.181207321Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:57:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:35.182368236Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:57:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:35.182533679Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:57:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:35.776888 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:35.779972 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:35.783257 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:35.786386 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:35.788634 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:35 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:57:35.788728 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:57:36 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:36.583212097Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:36.834973 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:57:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:36.835001 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:36.838181 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:36.859290 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:36.897190 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:36.962120 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:36.962272 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:57:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:37.058176 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:57:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:37.460810 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:57:38 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:38.595130206Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:40.182999678Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:57:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:40.200545861Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:57:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:40.606057520Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:42.617871352Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:44 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:44.698238333Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:45.205382828Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:57:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:46.709848850Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:48 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:48.732652284Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:57:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:50.824183258Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:57:50 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:57:50.831466 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:57:50 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:57:50.831532 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:57:50 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:57:50.831534 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:57:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:55.827342994Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:57:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:55.828259949Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:57:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:55.828407873Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:57:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:56.356599 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:56.360030 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:56.362884 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:56.365303 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:56.367675 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:57:56 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:57:56.367779 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:57:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:57.022161 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:57.025893 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:57.043126 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:57.054590 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:57.065984 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:57.066081 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:57:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:57.176538 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:57:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:57.386124 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:57:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:57.596047 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:57:57 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:57.868999166Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:57:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:57:57.999016 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:57:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:57:59.893465703Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:00.828777802Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:58:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:00.848873152Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:58:01 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:01.904964639Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:03 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:03.916165039Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:05 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:05.853509675Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:58:05 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:05.927820339Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:07.941517192Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:09.956533935Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:11 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:11.516164149Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:58:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:16.516699960Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:58:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:16.517848283Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:58:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:16.518021763Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:58:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:16.873274 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:16.877402 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:16.879846 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:16.882166 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:16.884530 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:16 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:58:16.884630 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:58:17 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:17.917335 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:58:17 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:17.998426356Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:58:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:18.098320 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:58:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:18.107753 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:58:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:18.109838 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:58:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:18.122030 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:58:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:18.135435 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:58:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:18.148824 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:58:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:18.149680 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:58:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:18.740012 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: the server is currently unable to handle the request (get endpointslices.discovery.k8s.io)
Oct 29 02:58:19 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:19.137309 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:58:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:20.012363134Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:21 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:21.518455483Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:58:21 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:21.537177351Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:58:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:22.026303644Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:22 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:58:22.667413 3222 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Oct 29 02:58:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:24.041113081Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:26 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:26.056868760Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:26 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:26.542380520Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:58:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:28.069374142Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:30.081580978Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:32 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:32.144990881Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:32 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:32.348431026Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:58:32 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:58:32.357067 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:58:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:37.349221298Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:58:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:37.350158068Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:58:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:37.350316715Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:58:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:37.704453 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:37.708529 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:37.711708 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:37.713999 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:37.716216 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:37 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:58:37.716315 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:58:38 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:38.734480 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:58:38 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:38.937710 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:58:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:39.018397 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:58:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:39.212471 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:58:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:39.216028 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:58:39 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:39.221115813Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:58:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:39.241899 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:58:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:39.258580 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:58:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:39.272932 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:58:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:39.273049 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:58:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:39.355068 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:58:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:39.403866 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: the server is currently unable to handle the request (get runtimeclasses.node.k8s.io)
Oct 29 02:58:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:41.246800984Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:42.351135610Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:58:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:42.368069386Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:58:43 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:43.259205924Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:45.269255219Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:47.279716235Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:47.372828324Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:58:49 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:49.326589561Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:51.336652415Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:53 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:53.369312849Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:58:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:54.095475027Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:58:54 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:58:54.103748 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:58:54 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:58:54.103749 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:58:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:59.095818742Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:58:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:59.097748669Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:58:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:58:59.097916112Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:58:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:59.472850 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:59.474780 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:59.477207 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:59.479166 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:58:59.481135 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:58:59 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:58:59.481227 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:59:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:00.353054 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:00.355317 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:00.368912 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:00.381827 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:00.402157 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:00.402867 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:59:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:00.411223291Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:00.517310 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:59:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:00.722678 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:59:01 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:01.072960 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:59:01 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:01.126685 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:59:01 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:01.930242 3222 controller.go:144] failed to ensure lease exists, will retry in 1.6s, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:59:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:02.425283064Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:02 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:02.859622 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
Oct 29 02:59:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:04.100071246Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:59:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:04.119992111Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:59:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:04.440033625Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:06.455330620Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:08 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:08.472055341Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:09.124809576Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:59:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:10.481680488Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:12.493563703Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:14.546214192Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:15 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:15.763758813Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:59:15 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:59:15.773789 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:59:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:20.765666825Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:59:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:20.766568410Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:59:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:20.766728057Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:59:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:20.970517 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:59:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:20.974133 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:59:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:20.977672 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:59:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:20.980389 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:59:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:20.982329 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:59:20 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:59:20.982412 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:59:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:22.007387 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:59:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:22.009815 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 02:59:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:22.019692 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: the server is currently unable to handle the request (get runtimeclasses.node.k8s.io)
Oct 29 02:59:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:22.210744 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:59:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:22.602370942Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:22.617561 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:59:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:22.846156 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:22.848331 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:22.859393 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:22.870914 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:22.882880 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:22.882987 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:59:23 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:23.430551 3222 controller.go:144] failed to ensure lease exists, will retry in 1.6s, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:59:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:24.616626557Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:25.766984300Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:59:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:25.786351802Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:59:26 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:26.626961901Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:28.638929165Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:30.654813647Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:30.791341531Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:59:32 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:32.665897196Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:34 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:34.678240779Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:36 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:59:36.676877 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:59:36 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:36.723487889Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:59:36 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:59:36.732517 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 02:59:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:41.724570136Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:59:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:41.725737996Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:59:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:41.725899328Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:42.479062 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:42.482217 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:42.485468 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:42.488669 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:42.490958 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: I1029 02:59:42.491052 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:42.772008442Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:42.936207 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:42.938230 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:42.959295 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:42.980190 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:42.994381 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 02:59:42 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:42.994501 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 02:59:43 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:43.517733 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:59:43 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:43.727993 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:59:44 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:44.131076 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 02:59:44 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:44.783997562Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:46.726804396Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 02:59:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:46.749245245Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:59:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:46.799738343Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:48 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:48.813801790Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:49 ubuntu-k3s-wkr1 k3s[3222]: E1029 02:59:49.789293 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
Oct 29 02:59:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:50.828479867Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:51.754631941Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 02:59:52 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:52.901496686Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:54.911764772Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:57 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:57.110795872Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 02:59:57 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T02:59:57.319961750Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 02:59:57 ubuntu-k3s-wkr1 k3s[3222]: W1029 02:59:57.329513 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:00:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:02.320804720Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:00:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:02.321721861Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:00:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:02.321872026Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:00:03 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:03.238665 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:00:03 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:03.242182 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:00:03 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:03.245667 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:00:03 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:03.248882 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:00:03 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:03.251123 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:00:03 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:00:03.251218 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:00:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:04.045699 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:04.048751 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:04.060834 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:04.077489 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:04.094106 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:04.094212 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:00:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:04.170421714Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:04.268649 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:04.471984 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:04.846324 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 03:00:04 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:04.874682 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:06.183852487Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:07.322326315Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:00:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:07.339779727Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:00:08 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:08.195485350Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:10.206634090Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:12.244285854Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:12.356274768Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:00:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:14.255206801Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:16.269101669Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:18.307400171Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:19 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:19.119825193Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:00:19 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:00:19.128280 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:00:19 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:00:19.128795 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:00:19 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:00:19.128796 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:00:19 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:00:19.129771 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:00:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:24.120732490Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:00:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:24.121766593Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:00:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:24.121933480Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:00:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:25.170224 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:25.172221 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:25.184616 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:25.197719 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:25.214114 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:25 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:00:25.214486 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:00:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:25.248197 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:25.347110012Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:25.451372 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:25 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:25.854207 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:26.143543 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:26.148641 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:26.168532 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:26.188028 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:26.202311 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:26.202416 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:00:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:26.670048 3222 controller.go:144] failed to ensure lease exists, will retry in 1.6s, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:27 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:27.373629563Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:29.122570796Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:00:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:29.140777055Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:00:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:29.386707593Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:30 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:30.564999 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
Oct 29 03:00:31 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:31.400492004Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:33 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:33.414296394Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:34 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:34.148363975Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:00:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:35.534288256Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:37.545287518Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:39 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:39.641610290Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:40.023428424Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:00:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:45.023870914Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:00:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:45.024922442Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:00:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:45.025082552Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:00:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:45.771523 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:00:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:45.775146 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:00:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:45.777901 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:00:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:45.780727 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:00:45 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:45.783510 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:00:45 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:00:45.783629 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:00:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:46.249377 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:46.256392 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:46.268676 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:46.280189 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:46.291846 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:46.291936 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:00:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:46.677747184Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:00:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:46.801616 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:47.005159 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:47.081401 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: the server is currently unable to handle the request (get csidrivers.storage.k8s.io)
Oct 29 03:00:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:00:47.408728 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:00:48 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:48.691674489Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:50.026163777Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:00:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:50.048439469Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:00:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:50.707792994Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:52 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:52.720716114Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:54.735874249Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:55.052368317Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:00:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:56.750974287Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:00:58 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:00:58.762911697Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:00.226112260Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:01:00 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:01:00.238521 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:01:00 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:01:00.238521 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:01:05 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:05.228347534Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:01:05 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:05.229266027Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:01:05 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:05.229421118Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.192809 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: the server is currently unable to handle the request (get nodes)
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.329154 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.331303 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.343027 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.357289 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.378054 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.378168 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.400126 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.552772 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.555062 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.566277 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.577748 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.588959 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:01:06.589320 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.599874 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:06.802733424Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:06 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:06.804748 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:07.209658 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:07 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:07.496919 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: the server is currently unable to handle the request (get endpointslices.discovery.k8s.io)
Oct 29 03:01:08 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:08.818077317Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:10.230244316Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:01:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:10.250492469Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:01:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:10.828468910Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:12.840540595Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:14.852488922Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:15 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:15.255295897Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:01:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:16.865507753Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:18.876979293Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:20.779317990Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:01:20 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:01:20.784276 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:01:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:25.779786213Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:01:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:25.781442122Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:01:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:25.781614007Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:01:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:26.709996 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:01:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:26.714031 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:01:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:26.718104 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:01:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:26.720952 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:01:26 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:26.723234 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:01:26 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:01:26.723333 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:01:26 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:26.917928228Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:27 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:27.442562 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:27 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:27.451822 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:27 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:27.465299 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:27 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:27.488517 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:27 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:27.503025 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:27 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:27.503786 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:01:27 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:27.745403 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:27 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:27.948308 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:28 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:28.350864 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:28.943200930Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:30.782504271Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:01:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:30.800496669Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:01:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:30.957387152Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:32 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:32.971056316Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:34 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:34.988918190Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:35.804840499Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:01:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:37.007570960Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:39 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:39.018267267Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:41.102215765Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:41.483390963Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:01:41 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:01:41.493029 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:01:41 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:01:41.493029 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:01:41 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:01:41.493032 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:01:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:46.483578440Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:01:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:46.484779249Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:01:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:46.485022614Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:01:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:47.074530 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:01:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:47.078470 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:01:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:47.082012 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:01:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:47.083963 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:01:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:47.085994 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:01:47 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:01:47.086085 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:01:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:47.548021 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:47.550188 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:47.561520 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:47.580049 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:47.599515 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:47 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:47.599643 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:01:48 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:48.104864 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:48 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:48.142396242Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:01:48 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:48.233677 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 03:01:48 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:48.313458 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:48 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:48.716073 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:01:50 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:50.156468810Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:51.485630681Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:01:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:51.505390560Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:01:52 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:52.167726039Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:54.179282455Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:56.193598376Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:01:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:56.510706473Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:01:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:01:57.074574 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
Oct 29 03:01:58 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:01:58.282120273Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:00.295031870Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:02.352259095Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:02.964509065Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:02:02 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:02:02.979538 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:02:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:07.965343818Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:02:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:07.966518332Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:02:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:07.966688311Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:02:08 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:08.625798 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:08 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:08.629856 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:08 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:08.632713 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:08 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:08.635218 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:08 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:08.637962 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:08 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:02:08.638071 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:02:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:09.389479837Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:09 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:09.664196 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:09 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:09.678534 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:02:09 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:09.685431 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:09 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:09.712371 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:09 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:09.741164 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:09 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:09.774111 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:09 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:09.774803 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:02:09 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:09.901244 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:02:10 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:10.304635 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:02:11 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:11.401419497Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:12.967125880Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:02:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:12.988465431Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:02:13 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:13.420303538Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:15 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:15.432465648Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:17 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:17.445445680Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:17 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:17.995934903Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:02:19 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:19.460285056Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:21 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:21.116713 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Oct 29 03:02:21 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:21.525697631Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:23 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:23.536984688Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:25.637013360Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:25.883400321Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:02:25 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:02:25.892316 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:02:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:30.884640218Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:02:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:30.885550942Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:02:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:30.885698235Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:02:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:31.767423 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:31.771584 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:31.775291 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:31.778061 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:31 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:31.780494 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:31 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:02:31.780600 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:02:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:32.147219 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: the server is currently unable to handle the request (get csidrivers.storage.k8s.io)
Oct 29 03:02:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:32.409888 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: the server is currently unable to handle the request (get endpointslices.discovery.k8s.io)
Oct 29 03:02:32 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:32.696947208Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:32.796327 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:02:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:32.828303 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:32.831192 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:32.849078 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:32.861301 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:32.873729 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:32 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:32.873820 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:02:33 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:33.010707 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:02:33 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:33.413784 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:02:34 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:34.707622877Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:35.886549605Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:02:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:35.907742677Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:02:36 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:36.718285211Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:38 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:38.728468813Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:40 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:40.707715 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:kube-proxy" cannot list resource "services" in API group "" at the cluster scope
Oct 29 03:02:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:40.739354732Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:40.912151338Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:02:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:42.857102667Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:44 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:44.880951627Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:46 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:46.865700640Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:02:46 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:02:46.872019 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:02:46 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:02:46.872046 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:02:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:51.866082008Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:02:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:51.867090823Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:02:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:51.867236080Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:02:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:52.029747 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:52.033359 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:52.036769 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:52.039108 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:52.041299 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:02:52 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:02:52.041397 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:02:52 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:52.919038990Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:52.942041 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:52.944408 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:52.958547 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:52 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:52.983366 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:53 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:53.004464 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:02:53 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:53.005127 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:02:53 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:53.059613 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:02:53 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:53.265105 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:02:53 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:53.668133 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:02:54 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:02:54.471510 3222 controller.go:144] failed to ensure lease exists, will retry in 1.6s, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:02:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:54.931420913Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:56.868002018Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:02:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:56.887799168Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:02:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:56.941791509Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:02:58 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:02:58.952456382Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:00.963956635Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:01 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:01.891407526Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:03:03 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:03.041132910Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:05 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:05.050936294Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:07.177343308Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:07.644967981Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:03:07 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:03:07.654473 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:03:07 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:03:07.654492 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:03:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:12.645760800Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:03:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:12.646744857Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:03:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:12.646898280Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:03:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:13.301449 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:13.305061 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:13.308131 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:13.310717 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:13 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:13.313277 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:13 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:03:13.313390 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:03:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:14.057349 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:14.059775 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:14.070551 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:14.081760 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:14.094665 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:14.094778 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:03:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:14.217777533Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:14.365640 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:03:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:14.570058 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:03:14 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:14.973308 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:03:16 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:16.233273240Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:17 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:17.648040621Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:03:17 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:17.668393127Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:03:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:18.248402431Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:20 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:20.260442780Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:22.242533 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:kube-proxy" cannot list resource "services" in API group "" at the cluster scope
Oct 29 03:03:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:22.273356035Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:22 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:03:22.666256 3222 sysinfo.go:203] Nodes topology is not available, providing CPU topology
Oct 29 03:03:22 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:22.672699448Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:03:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:24.283732122Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:26 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:26.294435084Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:28.345637418Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:28.515520954Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:03:28 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:03:28.527200 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:03:28 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:03:28.527262 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:03:33 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:33.516845986Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:03:33 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:33.517895968Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:03:33 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:33.518068336Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:03:34 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:34.069254 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:34 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:34.073191 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:34 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:34.076782 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:34 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:34.078981 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:34 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:34.081133 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:34 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:03:34.081232 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:03:34 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:34.771600 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: the server is currently unable to handle the request (get endpointslices.discovery.k8s.io)
Oct 29 03:03:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:35.099236 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:03:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:35.134658 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:35.136472 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:35.146231 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:35.157875 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:35.170191 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:35.170285 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:03:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:35.327681 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:03:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:35.407725838Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:35 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:35.730718 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:03:37 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:37.419200758Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:38 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:38.518671807Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:03:38 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:38.537093746Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:03:39 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:39.428479977Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:41.437928049Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:43 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:43.448179166Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:43 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:43.548344390Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:03:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:45.458009551Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:47.481033204Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:49 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:49.459424873Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:03:49 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:03:49.471467 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:03:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:54.460426559Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:03:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:54.461558374Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:03:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:54.461809778Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:03:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:55.108769 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:55.112618 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:55.115093 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:55.117470 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:55 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:55.119923 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:03:55 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:03:55.120031 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:03:55 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:55.522306372Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:56.171497 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:03:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:56.223499 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:56.234368 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:56.257396 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:56.270253 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:56.285223 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:03:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:56.285343 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:03:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:56.409411 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:03:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:56.570966 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: the server is currently unable to handle the request (get runtimeclasses.node.k8s.io)
Oct 29 03:03:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:56.714317 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 03:03:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:03:56.822247 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:03:57 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:57.535452825Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:03:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:59.462963185Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:03:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:59.478646733Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:03:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:03:59.546195331Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:01 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:01.557087329Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:03 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:03.569108859Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:04.483170726Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:04:05 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:05.609846357Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:07 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:07.637727162Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:09.555972909Z" level=error msg="Remotedialer proxy error" error="read tcp 192.168.86.91:60000->192.168.86.90:6443: read: connection reset by peer"
Oct 29 03:04:09 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:04:09.563850 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:04:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:14.556445617Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:04:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:14.557507360Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:04:14 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:14.557658636Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:04:15 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:15.681406246Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:15.701214 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:04:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:15.703090 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:04:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:15.739398 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:04:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:15.773962 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:04:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:15.788658 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:04:15 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:04:15.789105 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:04:15 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:15.804098 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:04:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:16.006771 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:04:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:16.196732 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: the server is currently unable to handle the request (get nodes)
Oct 29 03:04:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:16.343554 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:16.346072 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:16.366691 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:16.410136 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:16.433587 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:16 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:16.433696 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:04:17 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:17.696243411Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:19 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:19.558105573Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:04:19 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:19.583782291Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:04:19 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:19.709821564Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:21 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:21.723342483Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:22 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:22.102970 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
Oct 29 03:04:23 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:23.735757405Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:24 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:24.589025338Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:04:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:25.803767210Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:27 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:27.816491915Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:29.925232656Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:30 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:04:30.315009 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:04:30 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:30.351462033Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:04:30 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:04:30.352259 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:04:30 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:04:30.360656 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:04:30 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:04:30.360666 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:04:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:35.352343539Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:04:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:35.353392264Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:04:35 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:35.353679241Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:36.026593 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:36.030603 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:36.034248 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:36.036504 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:36.038716 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:04:36.038822 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:36.504754 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:36.508185 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:36.553670 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:36.567003 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:36.579473 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:36.579566 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:04:36 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:36.964282585Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:37.058154 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:04:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:37.260988 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:04:37 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:37.664476 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:04:38 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:38.978312714Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:40.354547633Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:04:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:40.373755181Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:04:40 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:40.988795561Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:42 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:42.999776064Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:45.011386851Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:45.378591372Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:04:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:47.028120392Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:49 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:49.042265263Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:04:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:51.088772842Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:04:51 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:04:51.095944 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:04:51 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:04:51.096230 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:04:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:56.090257761Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:04:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:56.091261320Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:04:56 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:56.091419670Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:04:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:56.870400 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:04:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:56.874532 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:04:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:56.878466 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:04:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:56.880742 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:04:56 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:56.883160 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:04:56 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:04:56.883268 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:04:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:57.412863 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: the server is currently unable to handle the request (get endpointslices.discovery.k8s.io)
Oct 29 03:04:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:57.636748 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:57.638545 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:57.649259 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:57.660156 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:57.671483 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:57.671573 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:04:57 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:57.911812 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:04:58 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:58.117643 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:04:58 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:04:58.150666130Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:04:58 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:04:58.521229 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:00.173335761Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:01 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:01.092415898Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:05:01 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:01.112889585Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:05:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:02.184031851Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:04.194430880Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:06.122497129Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:05:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:06.217327775Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:08 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:08.228282850Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:10.239321394Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:12.479151945Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:13 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:13.059635635Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:05:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:18.061095783Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:05:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:18.062748871Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:05:18 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:18.062893499Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:05:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:18.808789 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:05:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:18.812225 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:05:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:18.815252 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:05:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:18.818338 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:05:18 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:18.821185 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:05:18 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:05:18.821284 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:05:19 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:19.513324456Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:05:19 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:19.719727 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:05:19 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:19.722061 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:05:19 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:19.737947 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:05:19 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:19.756371 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:05:19 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:19.771824 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:05:19 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:19.772874 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:05:19 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:19.855559 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:19 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:19.989978 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: the server is currently unable to handle the request (get services)
Oct 29 03:05:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:20.057882 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:20.119525 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: the server is currently unable to handle the request (get runtimeclasses.node.k8s.io)
Oct 29 03:05:20 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:20.460475 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:21 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:21.527126367Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:23 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:23.063438847Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:05:23 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:23.083960654Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:05:23 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:23.538405427Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:25 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:25.549354438Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:27 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:27.560688723Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:28 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:28.093431927Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:05:29 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:29.572119433Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:31 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:31.583556514Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:33 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:33.055068211Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:05:33 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:05:33.071298 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Node ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:05:33 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:05:33.071298 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:05:33 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:05:33.071345 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.CSIDriver ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:05:38 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:38.056244347Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:05:38 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:38.057276832Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:05:38 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:38.057424571Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.272302 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.274280 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.285538 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.296946 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.309802 3222 controller.go:187] failed to update lease, error: the server is currently unable to handle the request (put leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:05:39.310194 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.322256 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.525769 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:39.623376616Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.813098 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.822663 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.849091 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.872309 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.897559 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.897680 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:05:39 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:39.941620 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:40 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:40.744934 3222 controller.go:144] failed to ensure lease exists, will retry in 1.6s, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:05:41 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:41.634741062Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:43 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:43.057679535Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:05:43 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:43.118143492Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:05:43 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:43.646214652Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:45 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:45.656817619Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:46 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:46.497899 3222 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: endpointslices.discovery.k8s.io is forbidden: User "system:kube-proxy" cannot list resource "endpointslices" in API group "discovery.k8s.io" at the cluster scope
Oct 29 03:05:47 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:47.667974501Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:48 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:48.124024395Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:05:49 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:49.685550698Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:51 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:51.696265484Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:53 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:53.774422581Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:05:54 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:54.131996064Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Oct 29 03:05:54 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:05:54.141660 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.RuntimeClass ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:05:54 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:05:54.141757 3222 reflector.go:436] k8s.io/client-go/informers/factory.go:134: watch of *v1.Service ended with: very short watch: k8s.io/client-go/informers/factory.go:134: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:05:54 ubuntu-k3s-wkr1 k3s[3222]: W1029 03:05:54.142242 3222 reflector.go:436] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: watch of *v1.Pod ended with: very short watch: k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:66: Unexpected watch close - watch lasted less than a second and no items received
Oct 29 03:05:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:59.133572959Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:05:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:59.134851849Z" level=error msg="Failed to connect to proxy" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:05:59 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:05:59.135018884Z" level=error msg="Remotedialer proxy error" error="dial tcp 192.168.86.90:6443: connect: connection refused"
Oct 29 03:05:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:59.558621 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:05:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:59.561861 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:05:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:59.564933 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:05:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:59.568184 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:05:59 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:05:59.570442 3222 controller.go:187] failed to update lease, error: Put "https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ubuntu-k3s-wkr1?timeout=10s": EOF
Oct 29 03:05:59 ubuntu-k3s-wkr1 k3s[3222]: I1029 03:05:59.570543 3222 controller.go:114] failed to update lease using latest lease, fallback to ensure lease, err: failed 5 attempts to update lease
Oct 29 03:06:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:06:00.586626 3222 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:06:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:06:00.790546 3222 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:06:00 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:06:00.808757224Z" level=info msg="Waiting for node ubuntu-k3s-wkr1: the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:06:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:06:00.944854 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:06:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:06:00.950617 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:06:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:06:00.968006 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:06:00 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:06:00.991962 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:06:01 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:06:01.014910 3222 kubelet_node_status.go:470] "Error updating node status, will retry" err="error getting node \"ubuntu-k3s-wkr1\": the server is currently unable to handle the request (get nodes ubuntu-k3s-wkr1)"
Oct 29 03:06:01 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:06:01.015012 3222 kubelet_node_status.go:457] "Unable to update node status" err="update node status exceeds retry count"
Oct 29 03:06:01 ubuntu-k3s-wkr1 k3s[3222]: E1029 03:06:01.205247 3222 controller.go:144] failed to ensure lease exists, will retry in 800ms, error: the server is currently unable to handle the request (get leases.coordination.k8s.io ubuntu-k3s-wkr1)
Oct 29 03:06:02 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:06:02.821381966Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:06:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:06:04.135980307Z" level=info msg="Connecting to proxy" url="wss://192.168.86.90:6443/v1-k3s/connect"
Oct 29 03:06:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:06:04.155471030Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:06:04 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:06:04.831585324Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:06:06 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:06:06.842062207Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:06:08 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:06:08.893217822Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:06:09 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:06:09.160177161Z" level=warning msg="Unable to watch for tunnel endpoints: unknown (get endpoints)"
Oct 29 03:06:10 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:06:10.905846528Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Oct 29 03:06:12 ubuntu-k3s-wkr1 k3s[3222]: time="2021-10-29T03:06:12.916518842Z" level=info msg="Waiting for node ubuntu-k3s-wkr1 CIDR not assigned yet"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment