cgroups_are_enabled_though.log

filbot@ubuntu-k3s-master:~$ cat /boot/firmware/cmdline.txt 
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

k3s-master_cgroups_not_set.log

Nov 01 19:47:48 ubuntu-k3s-master systemd[1]: k3s.service: Consumed 4.933s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit k3s.service completed and consumed the indicated resources.
Nov 01 19:47:48 ubuntu-k3s-master systemd[1]: Starting Lightweight Kubernetes...
░░ Subject: A start job for unit k3s.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit k3s.service has begun execution.
░░
░░ The job identifier is 3372804.
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:49.541569833Z" level=info msg="Starting k3s v1.19.15+k3s2 (0fbbc5af)"
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:49.542266526Z" level=info msg="Cluster bootstrap already complete"
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:49.598965767Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:49.599096506Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:49.599552110Z" level=info msg="Database tables and indexes are up to date"
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:49.604475255Z" level=info msg="Kine listening on unix://kine.sock"
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:49.605072616Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=unknown --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --enable-admission-plugins=NodeRestriction --etcd-servers=unix://kine.sock --insecure-port=0 --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=k3s --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: I1101 19:47:49.610974 1789143 server.go:652] external host was not specified, using 192.168.86.90
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: I1101 19:47:49.613135 1789143 server.go:177] Version: v1.19.15+k3s2
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: I1101 19:47:49.634677 1789143 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: I1101 19:47:49.634763 1789143 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: I1101 19:47:49.638434 1789143 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: I1101 19:47:49.638520 1789143 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
Nov 01 19:47:49 ubuntu-k3s-master k3s[1789143]: I1101 19:47:49.691911 1789143 master.go:271] Using reconciler: lease
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: W1101 19:47:51.006112 1789143 genericapiserver.go:418] Skipping API batch/v2alpha1 because it has no resources.
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: W1101 19:47:51.069152 1789143 genericapiserver.go:418] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: W1101 19:47:51.140129 1789143 genericapiserver.go:418] Skipping API node.k8s.io/v1alpha1 because it has no resources.
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: W1101 19:47:51.242469 1789143 genericapiserver.go:418] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: W1101 19:47:51.269185 1789143 genericapiserver.go:418] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: W1101 19:47:51.326513 1789143 genericapiserver.go:418] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: W1101 19:47:51.392801 1789143 genericapiserver.go:418] Skipping API apps/v1beta2 because it has no resources.
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: W1101 19:47:51.392977 1789143 genericapiserver.go:418] Skipping API apps/v1beta1 because it has no resources.
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: I1101 19:47:51.431269 1789143 plugins.go:158] Loaded 12 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: I1101 19:47:51.431380 1789143 plugins.go:161] Loaded 10 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:51.483538076Z" level=info msg="Waiting for API server to become available"
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:51.483580150Z" level=info msg="Running kube-scheduler --address=127.0.0.1 --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --port=10251 --profiling=false --secure-port=0"
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: I1101 19:47:51.484493 1789143 registry.go:173] Registering SelectorSpread plugin
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: I1101 19:47:51.484562 1789143 registry.go:173] Registering SelectorSpread plugin
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:51.485282456Z" level=info msg="Running kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --port=10252 --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=0 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --use-service-account-credentials=true"
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:51.489905366Z" level=info msg="Node token is available at /var/lib/rancher/k3s/server/token"
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:51.490036308Z" level=info msg="To join node to cluster: k3s agent -s https://192.168.86.90:6443 -t ${NODE_TOKEN}"
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:51.493616215Z" level=info msg="Wrote kubeconfig /etc/rancher/k3s/k3s.yaml"
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:51.493753120Z" level=info msg="Run: k3s kubectl"
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:51.494087300Z" level=warning msg="Failed to find cpuset cgroup, you may need to add \"cgroup_enable=cpuset\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:51.494187076Z" level=error msg="Failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
Nov 01 19:47:51 ubuntu-k3s-master k3s[1789143]: time="2021-11-01T19:47:51.494270149Z" level=fatal msg="failed to find memory cgroup, you may need to add \"cgroup_memory=1 cgroup_enable=memory\" to your linux cmdline (/boot/cmdline.txt on a Raspberry Pi)"
Nov 01 19:47:51 ubuntu-k3s-master systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit k3s.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Nov 01 19:47:51 ubuntu-k3s-master systemd[1]: k3s.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit k3s.service has entered the 'failed' state with result 'exit-code'.
Nov 01 19:47:51 ubuntu-k3s-master systemd[1]: Failed to start Lightweight Kubernetes.
░░ Subject: A start job for unit k3s.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit k3s.service has finished with a failure.
░░
░░ The job identifier is 3372804 and the job result is failed.
Nov 01 19:47:51 ubuntu-k3s-master systemd[1]: k3s.service: Consumed 5.226s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit k3s.service completed and consumed the indicated resources.
Nov 01 19:47:56 ubuntu-k3s-master systemd[1]: k3s.service: Scheduled restart job, restart counter is at 40536.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ Automatic restarting of the unit k3s.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Nov 01 19:47:56 ubuntu-k3s-master systemd[1]: Stopped Lightweight Kubernetes.
░░ Subject: A stop job for unit k3s.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit k3s.service has finished.
░░
░░ The job identifier is 3372887 and the job result is done.
Nov 01 19:47:56 ubuntu-k3s-master systemd[1]: k3s.service: Consumed 5.226s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit k3s.service completed and consumed the indicated resources.
Nov 01 19:47:56 ubuntu-k3s-master systemd[1]: Starting Lightweight Kubernetes...
░░ Subject: A start job for unit k3s.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit k3s.service has begun execution.
░░
░░ The job identifier is 3372887.
Nov 01 19:47:57 ubuntu-k3s-master k3s[1789189]: time="2021-11-01T19:47:57.303463172Z" level=info msg="Starting k3s v1.19.15+k3s2 (0fbbc5af)"
Nov 01 19:47:57 ubuntu-k3s-master k3s[1789189]: time="2021-11-01T19:47:57.304116310Z" level=info msg="Cluster bootstrap already complete"