Update Kubernetes... Not Flatcar.

README.md

This is a demo of updating Kubernetes version on Flatcar without updating the OS leveraging Systemd sysext and Kured.

notes:

• It has been tested on a simple control-plane node on Qemu but can be easily extended to CAPI.
• The kubernetes sysext image is provided by https://github.com/flatcar/sysext-bakery.git

Boot an instance with the following Butane config.yaml:

variant: flatcar
version: 1.0.0
systemd:
  units:
    - name: [email protected]
    - name: [email protected]
      enabled: true
    - name: [email protected]
      enabled: true
    - name: [email protected]
      contents: |
        [Unit]
        Description=Bootstrap the %I component
        ConditionPathIsSymbolicLink=!/etc/extensions/%I.raw
        ConditionVirtualization=!container
        DefaultDependencies=no

        Before=systemd-sysext.service

        [Service]
        Type=oneshot
        RemainAfterExit=yes
        ExecStart=/usr/lib/systemd/systemd-sysupdate --component=%I update

        [Install]
        WantedBy=sysinit.target
    - name: [email protected]
      contents: |
        [Unit]
        Description=Automatic System Update
        Documentation=man:systemd-sysupdate.service(8)
        
        # For containers we assume that the manager will handle updates. And we likely
        # can't even access our backing block device anyway.
        ConditionVirtualization=!container
        
        [Timer]
        # Trigger the update 15min after boot, and then – on average – every 6h, but
        # randomly distributed in a 2h…6h interval. In addition trigger things
        # persistently once on each Saturday, to ensure that even on systems that are
        # never booted up for long we have a chance to to do the update.
        OnBootSec=15min
        OnUnitActiveSec=2h
        OnCalendar=Sat
        RandomizedDelaySec=4h
        Persistent=yes
        
        [Install]
        WantedBy=timers.target
    - name: [email protected]
      contents: |
        [Unit]
        Description=Automatic System Update
        Documentation=man:systemd-sysupdate.service(8)
        Wants=network-online.target
        After=network-online.target
        ConditionVirtualization=!container
        
        [Service]
        Type=simple
        NotifyAccess=main
        ExecStart=/usr/lib/systemd/systemd-sysupdate --component=%I update
        ExecStartPost=/usr/bin/touch /run/reboot-required
        CapabilityBoundingSet=CAP_CHOWN CAP_FOWNER CAP_FSETID CAP_MKNOD CAP_SETFCAP CAP_SYS_ADMIN CAP_SETPCAP CAP_DAC_OVERRIDE CAP_LINUX_IMMUTABLE
        NoNewPrivileges=yes
        MemoryDenyWriteExecute=yes
        ProtectHostname=yes
        RestrictRealtime=yes
        RestrictNamespaces=net
        RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
        SystemCallFilter=@system-service @mount
        SystemCallErrorNumber=EPERM
        SystemCallArchitectures=native
        LockPersonality=yes
        
        [Install]
        Also=systemd-sysupdate.timer
storage:
  files:
    - path: /etc/sysupdate.kubernetes.d/kubernetes.conf
      contents:
        inline: |
          [Transfer]
          Verify=false
          
          [Source]
          Type=url-file
          Path=http://192.168.1.17:8000/
          [email protected]
          
          [Target]
          InstancesMax=3
          Type=regular-file
          Path=/opt/extensions/kubernetes
          CurrentSymlink=/etc/extensions/kubernetes.raw

note: Don't forget to update the IP/URL of the systemd-sysext source. For CAPI, I use GCS Bucket for example.

SSH on the instance and assert everything works fine and start the Kubernetes control plane:

$ systemd-sysext status
HIERARCHY EXTENSIONS SINCE
/opt      none       -
/usr      kubernetes Fri 2023-07-28 08:55:01 UTC
          oem-qemu
$ ls -l /etc/extensions/
total 0
lrwxrwxrwx. 1 root root 37 Jul 28 08:54 kubernetes.raw -> /opt/extensions/kubernetes-1.26.6.raw
lrwxrwxrwx. 1 root root 32 Jul 28 08:54 oem-qemu.raw -> /oem/sysext/oem-qemu-initial.raw
$ sudo kubeadm init
...
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ kubectl apply -f https://docs.projectcalico.org/archive/v3.23/manifests/calico.yaml

Install Kured to manage the reboot mechanism when there is a new update:

$ latest=$(curl -s https://api.github.com/repos/kubereboot/kured/releases | jq -r '.[0].tag_name')
$ curl -O -fsSL "https://github.com/kubereboot/kured/releases/download/$latest/kured-$latest-dockerhub.yaml"
# edit kured-$latest-dockerhub.yaml to have a --period to 10s for demo purposes
$ kubectl apply -f kured-$latest-dockerhub.yaml

Force the update for demo purposes (otherwise it's a regular systemd timer). The node is running version 1.26.6 before the update:

$ kubectl get nodes
NAME        STATUS   ROLES           AGE     VERSION
localhost   Ready    control-plane   6m57s   v1.26.6
$ sudo systemctl start systemd-sysupdate
$ journalctl -f -u systemd-sysupdate
...
Jul 28 09:05:20 localhost systemd-sysupdate[5895]: Updated symlink '/etc/extensions/kubernetes.raw' → '../../opt/extensions/kubernetes-1.27.3.raw'.
Broadcast message from root@localhost (Fri 2023-07-28 09:20:12 UTC):

The system will reboot now!

Connection to 127.0.0.1 closed by remote host.
Connection to 127.0.0.1 closed.

You can SSH back on the node and assert that Kubernetes has been upgraded:

$ kubectl get nodes
NAME        STATUS   ROLES           AGE   VERSION
localhost   Ready    control-plane   13m   v1.27.3

Maybe better enable systemd-sysupdate.timer instead of the service (With Also= being used in the service Install section the result should be the same as it already would enable the timer instead of the service).

@pothos thanks ! updated

I would also use a subfolder for the extensions, e.g., /opt/sysext/kubernetes/.

One can also define Kubernetes as a component, so that it can be updated independently instead of together with all other sysext images. This would be done through using /etc/sysupdate.kubernetes.d/kubernetes.conf and then a new service that runs /usr/lib/systemd/systemd-sysupdate --component=kubernetes update.

InstancesMax=3 is maybe also a good setting to prevent filling the disk over time.

And how about a helper service that runs systemd-sysupdate.service once before systemd-sysext.service under the condition that the symlink is not existing. That would allow us to remove the symlink and source download from Butane, and we get a config that does not hardcode any version.

@pothos thanks for the feedback. What do you think of the approach above? I think using a [email protected] allows to leverage this component concept of systemd-sysupdate.

Yes, looks good (the bootstrap is missing network-online.target).
We also need to find a way of only running touch /run/reboot-required if there was an update.