Installing microk8s in an LXC container

microk8s_in_lxc.md

Installing microk8s in an LXC container

I wanted to run Microk8s on a Proxmox 6 host inside of an LXC container. These are my notes from the journey.

1. Create a privileged LXC container through the Proxmox web interface

• Enable nesting and FUSE
  • In Proxmox UI, select container, then Options > Features > Check nesting and FUSE boxes

2. SSH into the Proxmox host and edit the container's config in /etc/pve/lxc/.conf
   • Add the following lines
     • lxc.apparmor.profile: unconfined
     • lxc.cap.drop:
     • lxc.mount.auto: proc:rw sys:rw
3. Start (or restart) the container
4. SSH into the container and create a symlink for /dev/kmsg, which is missing in Ubuntu 19.10 containers
   • ln -s /dev/console /dev/kmsg
   • Has to be repeated on container reboot, which is annoying.
5. Install snapd: apt install snapd
6. Install microk8s: snap install microk8s --classic

The snap commands may need to be run more than once to get past errors.

Troubleshooting

If you get "cannot change profile for the next exec call: No such file or directory", try running: apparmor_parser -r /var/lib/snapd/apparmor/profiles/*

@jez500 I'm glad that it was helpful! I haven't done anything to fix the symlink issue yet, but let me know if the rc.local solution works well for you. My plan was to create something with systemd or cron on the host that would periodically fix the container, which doesn't seem ideal.

@acj rc.local didn't seem to work, server rebooted over the last week and started getting the "cannot change profile" errors with the /dev/kmsg symlink missing. Again your notes helped me out :) I created this script which I might hook up to a cron in the future but for now it seems to do the trick

#!/bin/bash

# If this symlink is missing, likely microk8s isn't running
if [ ! -L /dev/kmsg ]; then
  ln -s /dev/console /dev/kmsg
  apparmor_parser -r /var/lib/snapd/apparmor/profiles/*
  microk8s stop
  microk8s start
fi

In Proxmox >7.0 (LXC container with Ubuntu 20.04.3 LTS) I had issue with below warning:
WARNING: The memory cgroup is not enabled. The cluster may not be functioning properly. Please ensure cgroups are enabled See for example: https://microk8s.io/docs/install-alternatives#heading--arm
In order to fix my issue I had to switch back from cgroupv2 to legacy solution.
Open shell of proxmox host and change GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub to below one
GRUB_CMDLINE_LINUX_DEFAULT="systemd.unified_cgroup_hierarchy=0 quiet"
then update-grub && reboot.
Docs:
https://pve.proxmox.com/pve-docs/chapter-pct.html#pct_cgroup
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_edit_kernel_cmdline

I found that if you put a @reboot line in your crontab it will add the ln when the container boots.

@reboot ln -s /dev/console /dev/kmsg

To add the line above, type crontab -e which will open your crontab in an editor, then add the line above to the bottom of the file.

@VV0JC13CH Your suggestion to edit grub helped in getting rid of the cgroup error, however I am still having problems.

When I run microk8s start I get the message Started., the terminal freezes for about a minute and then I get my prompt back.

When I run microk8s status I get

microk8s is not running. Use microk8s inspect for a deeper inspection.

I run microk8s inspect and get

microk8s is not running. Use microk8s inspect for a deeper inspection.
root@mk8s-3-1:~# microk8s inspect
Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-kubelite is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster
Inspecting juju
  Inspect Juju
Inspecting kubeflow
  Inspect Kubeflow

Building the report tarball
  Report tarball is at /var/snap/microk8s/2695/inspection-report-20211205_114227.tar.gz

microk8s kubectl get nodes I get

The connection to the server 127.0.0.1:16443 was refused - did you specify the right host or port?

If I run it again I get

NAME       STATUS     ROLES    AGE   VERSION
mk8s-3-1   NotReady   <none>   31m   v1.22.4-3+adc4115d990346

and then it switches between these two states, so looks like microk8s is trying to start, failing and then restarting

Can anyone think of anything I can try to get this working

A few updates to the original steps using Debian 11 LXC Template / June 2022

1. Create a privileged LXC container through the Proxmox web interface, for swap amount enter 0.

• Enable nesting and FUSE
  • In Proxmox UI, select container, then Options > Features > Check nesting and FUSE boxes

2. SSH into the Proxmox host and edit the container's config in /etc/pve/lxc/.conf
   • Add the following lines

lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file 0 0

3. Start (or restart) the container
4. Install snapd: apt install -y snapd squashfuse fuse / reboot!
5. Install microk8s: snap install microk8s --classic

A few updates to the original steps using Debian 11 LXC Template / June 2022

1. Create a privileged LXC container through the Proxmox web interface, for swap amount enter 0.

• Enable nesting and FUSE

  • In Proxmox UI, select container, then Options > Features > Check nesting and FUSE boxes

2. SSH into the Proxmox host and edit the container's config in /etc/pve/lxc/.conf

   • Add the following lines

lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file 0 0

3. Start (or restart) the container
4. Install snapd: apt install -y snapd squashfuse fuse / reboot!
5. Install microk8s: snap install microk8s --classic

I had to add lxc.mount.entry: /sys/kernel/security sys/kernel/security none bind,create=file 0 0 to /etc/pve/lxc/.conf and also had to run apt install sudo in the container before snap install microk8s --classic to get this to install.

I have made no considerations for security.

5. snap install microk8s --classic

it's works. thanks

Followed the instructions and got the microk8s cluster working, however pods do not have access to networking (e.g. socket: permission denied or similar when attempting to bind ports or use commands such as ip addr list). I can see pods are being allocated IPs by Calico correctly otherwise via get pod.

My guess is this is a capabilities issue but lxc.cap.drop: is set as mentioned above and I even tried launching pods with explicit securityContext policy but still no luck.

Did anyone run across this by chance? I tried Debian 11 and Ubuntu 22.04 templates but same result

I did not encounter this issue if launching as a VM instead of LXC

Worked on Pimox 7.2 (Proxmox for raspberry pi arm64) and Ubuntu 22.04 LXC. Two nodes installed without any issues, one was complaining about missing microk8s LXC profile; After couple tries and a reboot, snap managed to install package.

I got it working on Promox 8.0.3 with Debian 12 but had to change some steps:

1. Create a privileged Debian 12 LXC container through the Proxmox web interface, for swap amount enter 0.
2. SSH into the Proxmox host and edit the container's config in /etc/pve/lxc/<CT id>.conf and add the following lines:

features: fuse=1,nesting=1
lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
lxc.mount.entry: /sys/kernel/security sys/kernel/security none bind,create=file 0 0

3. Start the server and run crontab -e and add the line @reboot ln -s /dev/console /dev/kmsg
4. apt install -y snapd squashfuse fuse sudo && reboot - container will restart
5. snap install microk8s --classic

Thanks for the updated fix to be able to install microk8s on lxc in Debian 12. You saved me a lot of extra time I apreciate it !

I'm install on Proxmox 8.0.3, Debian 12 LXC with @nastynaz's steps, but I'm getting this error during snap install microk8s --classic

error: cannot perform the following tasks:
- Mount snap "microk8s" (6070) (umount: /snap/microk8s/6070: must be superuser to unmount.)
- Run install hook of "microk8s" snap if present (run hook "install": / not root-owned 100000:100000)

I'm install on Proxmox 8.0.3, Debian 12 LXC with @nastynaz's steps, but I'm getting this error during snap install microk8s --classic

error: cannot perform the following tasks:
- Mount snap "microk8s" (6070) (umount: /snap/microk8s/6070: must be superuser to unmount.)
- Run install hook of "microk8s" snap if present (run hook "install": / not root-owned 100000:100000)

I've just followed his hint literally and it worked as described!

`root@pve1:# lxc-attach 107
root@k8s:# snap install microk8s --classic
2023-10-28T00:59:20Z INFO Waiting for automatic snapd restart...
Warning: /snap/bin was not found in your $PATH. If you've not restarted your session since you
installed snapd, try doing that. Please see https://forum.snapcraft.io/t/9469 for more
details.

microk8s (1.27/stable) v1.27.6 from Canonical✓ installed`

You sure you unticked the unprivileged checkbox?

If I find time I'll create a playbook for that over the course of the WE ;-)

Here's what works for me:

`─➤ cat ansible/play_deploy_k8s_create-lxc.yml

• name: Setup LXC-Container (K8S) on PVE for deploying of K8S on it
  hosts: one-bn-pve1
  gather_facts: yes
  vars:
  ansible_python_interpreter: "/usr/bin/python3"

  username: "root@pam"
  user_password: ""
  lxc_root_password: "FOO"

  api_host: "10.10.xxx.yy"
  netif: '{"net0":"name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0"}'
  os_template_name: "debian-12-standard_12.2-1_amd64.tar.zst"

  vmid: 180
  node: pve1
  hostname: "k8s-{{ vmid }}"
  cpu_cores: 2
  memory: 1024
  rootfs_disk_size: "8"
  storage: "Ceph_1TB"

  module_defaults:
  ansible.builtin.get_url:
  validate_certs: false
  tasks:

  • name: Install Proxmoxer"
    ansible.builtin.apt:
    name: python3-proxmoxer
    state: latest
    update_cache: true

  • name: Check if LXC template is downloaded
    ansible.builtin.stat:
    path: "/var/lib/vz/template/cache/{{ os_template_name }}"
    register: p

  • ansible.builtin.debug: msg="Template was already downloaded. Skipping."
    when: p.stat.exists

  • name: Show IP
    ansible.builtin.shell: ip a

  • name: Download LXC template
    ansible.builtin.shell: pveam download local {{ os_template_name }}
    when: p.stat.exists == False

  • name: Create K8S LXC-Container
    connection: local
    community.general.proxmox:
    vmid: "{{ vmid }}"
    node: "{{ node }}"
    state: "present"

    purge: true

    onboot: true
    api_user: "{{ username }}"
    api_password: "{{ user_password }}"
    api_host: "{{ api_host }}"
    password: "{{ lxc_root_password }}"
    hostname: "{{ hostname }}-pve1"
    ostemplate: "local:vztmpl/{{ os_template_name }}"
    storage: "{{ storage }}"
    netif: "{{ netif }}"
    memory: "{{ memory }}"
    disk: "{{ rootfs_disk_size }}"
    swap: "0"
    unprivileged: false
    features:

    • fuse=1
    • nesting=1
      pubkey: ""
      register: container_present
      tags:
    • create

  • name: Check if K8S LXC is running
    ansible.builtin.shell: pct status {{ vmid }}
    register: lxc_running
    tags:

    • create
    • restart
    • lxc_started

  • name: Insert lines into LXC conf
    ansible.builtin.blockinfile:
    path: "/etc/pve/lxc/{{ vmid }}.conf"
    block: |
    lxc.apparmor.profile: unconfined
    lxc.cap.drop:
    lxc.mount.auto: proc:rw sys:rw
    lxc.mount.entry: /sys/kernel/security sys/kernel/security none bind,create=file 0 0
    backup: yes
    when: '"status: running" not in lxc_running.stdout'
    tags:

    • lxc_started
    • create
    • restart

  • name: Start K8S LXC container
    community.general.proxmox:
    vmid: "{{ vmid }}"
    node: "{{ node }}"
    state: "started"
    api_user: "{{ username }}"
    api_password: "{{ user_password }}"
    api_host: "{{ api_host }}"
    when: '"status: running" not in lxc_running.stdout'
    tags:

    • lxc_started
    • create
    • restart

  • name: get K8S LXC Container IP
    ansible.builtin.shell:
    cmd: pct exec {{ vmid }} ip addr show|grep 'inet 10'| awk -F ' ' '{print $2}'
    register: lxcip

  • name: show K8S LXC Container IP
    ansible.builtin.debug:
    msg: The Container {{ vmid }} has the following IP {{ lxcip.stdout }}
    when: lxcip.stdout != "" %`

`- hosts: "{{ machines }}"
#remote_user: "{{ remote_user }}"
tasks:

• name: Add CronJob "@reboot ln -s /dev/console /dev/kmsg"
  ansible.builtin.cron:
  name: "create /dev/kmsg"
  special_time: reboot
  job: "ln -s /dev/console /dev/kmsg"

• name: Install required Packages
  apt:
  pkg:

  • snapd
  • squashfuse
  • fuse
  • sudo
    state: latest
    update_cache: true

• name: Reboot
  ansible.builtin.reboot:
  msg: "Reboot initiated by Ansible"
  connect_timeout: 5
  reboot_timeout: 600
  pre_reboot_delay: 0
  post_reboot_delay: 30
  test_command: whoami

• name: Install microk8s
  community.general.snap:
  name: microk8s
  state: absent
  classic: true`

'- hosts: all

vars:
ansible_ssh_private_key_file: ~/.ssh/key
ansible_ssh_user: root

tasks:

• name: Write the LXCs host key to known_hosts
  connection: local
  shell: "ssh-keyscan -H {{ inventory_hostname }} >> ~/.ssh/known_hosts"

• name: Add CronJob "@reboot ln -s /dev/console /dev/kmsg"
  ansible.builtin.cron:
  name: "create /dev/kmsg"
  special_time: reboot
  job: "ln -s /dev/console /dev/kmsg"

• name: Install required Packages
  apt:
  pkg:

  • snapd
  • squashfuse
  • fuse
  • sudo
    state: latest
    update_cache: true

• name: Reboot
  ansible.builtin.reboot:
  msg: "Reboot initiated by Ansible"
  connect_timeout: 5
  reboot_timeout: 30
  pre_reboot_delay: 0
  post_reboot_delay: 10
  test_command: whoami

• name: Install microk8s
  community.general.snap:
  name: microk8s
  state: present
  classic: true'

Run them in the listed order and you'll have your 1st microk8s LXC

After two days of trying EVERYTHING, this is how i made it work on Proxmox 8.0.3:

1- Download debian 12 (debian-12-standard_12.2-1_amd64.tar.zst) ct template ( i don't know why, but i couldn't make it work with ubuntu ):

pveam download local debian-12-standard_12.2-1_amd64.tar.zst

2- Make a new CT with the following configurations:

• Make sure "Unprivileged Container" is unchecked, so it becomes privileged.
• Use downloaded debian template.
• Make sure sure swap is 0 in memory tab.
  NOTE: Don't start the CT yet.

3- On the created CT go to options tab, double click on "Features" and enable "Fuse" and "Nesting".
4- SSH into your Proxmox node and navigate to /etc/pve/lxc/, and open your {ct_id}.conf and add these few lines in:

lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
lxc.mount.entry: /sys/kernel/security sys/kernel/security none bind,create=file 0 0

5- Start the CT
6- Open your crontab using crontab -e and add this line at the end:

@reboot ln -s /dev/console /dev/kmsg

7- Install required apt packages:

apt update && apt upgrade -y && apt install snapd squashfuse fuse sudo -y

8- Reboot for changes to take effect.
9- After the reboot, finally you can install microk8s:

snap install microk8s --classic

10- Enjoy your first microk8s on Proxmox 🎉

perchance someone knows why I can't SSH in? "Permission denied, please try again." Everything else seems to work...

@stealthmachines, did you add your ssh keys or specified password when creating a container?

snap install microk8s --classic

Your original suggestion did not work for me in both debian and ubuntu but was a really good starting place.
The following conf works in both ubuntu and debian

root@ns26031989:~# cat /etc/pve/lxc/107.conf 
arch: amd64
cores: 2
features: mount=fuse,nesting=1
hostname: runUp
memory: 4096
net0: name=eth0,bridge=vmbr0,firewall=1,gw=149.202.93.254,hwaddr=02:00:00:3a:88:6e,ip=178.32.45.21/24,type=veth
ostype: debian
rootfs: local:107/vm-107-disk-0.raw,size=80G
swap: 0
lxc.apparmor.profile: unconfined
lxc.cap.drop: 
lxc.mount.auto: proc:rw sys:rw
lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file 0 0
lxc.mount.entry: /sys/kernel/security sys/kernel/security none bind,create=file 0 0

Excellent work, guys. You saved me hours....Thank you so much. I got one addtion to make, though:
It is important to assign static ip adresses to the container because due to some conflict in the configuration, dhcp does not seem to work anymore.