Installing microk8s in an LXC container

microk8s_in_lxc.md

Installing microk8s in an LXC container

I wanted to run Microk8s on a Proxmox 6 host inside of an LXC container. These are my notes from the journey.

1. Create a privileged LXC container through the Proxmox web interface

• Enable nesting and FUSE
  • In Proxmox UI, select container, then Options > Features > Check nesting and FUSE boxes

2. SSH into the Proxmox host and edit the container's config in /etc/pve/lxc/.conf
   • Add the following lines
     • lxc.apparmor.profile: unconfined
     • lxc.cap.drop:
     • lxc.mount.auto: proc:rw sys:rw
3. Start (or restart) the container
4. SSH into the container and create a symlink for /dev/kmsg, which is missing in Ubuntu 19.10 containers
   • ln -s /dev/console /dev/kmsg
   • Has to be repeated on container reboot, which is annoying.
5. Install snapd: apt install snapd
6. Install microk8s: snap install microk8s --classic

The snap commands may need to be run more than once to get past errors.

Troubleshooting

If you get "cannot change profile for the next exec call: No such file or directory", try running: apparmor_parser -r /var/lib/snapd/apparmor/profiles/*

Followed the instructions and got the microk8s cluster working, however pods do not have access to networking (e.g. socket: permission denied or similar when attempting to bind ports or use commands such as ip addr list). I can see pods are being allocated IPs by Calico correctly otherwise via get pod.

My guess is this is a capabilities issue but lxc.cap.drop: is set as mentioned above and I even tried launching pods with explicit securityContext policy but still no luck.

Did anyone run across this by chance? I tried Debian 11 and Ubuntu 22.04 templates but same result

I did not encounter this issue if launching as a VM instead of LXC

Worked on Pimox 7.2 (Proxmox for raspberry pi arm64) and Ubuntu 22.04 LXC. Two nodes installed without any issues, one was complaining about missing microk8s LXC profile; After couple tries and a reboot, snap managed to install package.

I got it working on Promox 8.0.3 with Debian 12 but had to change some steps:

1. Create a privileged Debian 12 LXC container through the Proxmox web interface, for swap amount enter 0.
2. SSH into the Proxmox host and edit the container's config in /etc/pve/lxc/<CT id>.conf and add the following lines:

features: fuse=1,nesting=1
lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
lxc.mount.entry: /sys/kernel/security sys/kernel/security none bind,create=file 0 0

3. Start the server and run crontab -e and add the line @reboot ln -s /dev/console /dev/kmsg
4. apt install -y snapd squashfuse fuse sudo && reboot - container will restart
5. snap install microk8s --classic

Thanks for the updated fix to be able to install microk8s on lxc in Debian 12. You saved me a lot of extra time I apreciate it !

I'm install on Proxmox 8.0.3, Debian 12 LXC with @nastynaz's steps, but I'm getting this error during snap install microk8s --classic

error: cannot perform the following tasks:
- Mount snap "microk8s" (6070) (umount: /snap/microk8s/6070: must be superuser to unmount.)
- Run install hook of "microk8s" snap if present (run hook "install": / not root-owned 100000:100000)

I'm install on Proxmox 8.0.3, Debian 12 LXC with @nastynaz's steps, but I'm getting this error during snap install microk8s --classic

error: cannot perform the following tasks:
- Mount snap "microk8s" (6070) (umount: /snap/microk8s/6070: must be superuser to unmount.)
- Run install hook of "microk8s" snap if present (run hook "install": / not root-owned 100000:100000)

I've just followed his hint literally and it worked as described!

`root@pve1:# lxc-attach 107
root@k8s:# snap install microk8s --classic
2023-10-28T00:59:20Z INFO Waiting for automatic snapd restart...
Warning: /snap/bin was not found in your $PATH. If you've not restarted your session since you
installed snapd, try doing that. Please see https://forum.snapcraft.io/t/9469 for more
details.

microk8s (1.27/stable) v1.27.6 from Canonical✓ installed`

You sure you unticked the unprivileged checkbox?

If I find time I'll create a playbook for that over the course of the WE ;-)

Here's what works for me:

`─➤ cat ansible/play_deploy_k8s_create-lxc.yml

• name: Setup LXC-Container (K8S) on PVE for deploying of K8S on it
  hosts: one-bn-pve1
  gather_facts: yes
  vars:
  ansible_python_interpreter: "/usr/bin/python3"

  username: "root@pam"
  user_password: ""
  lxc_root_password: "FOO"

  api_host: "10.10.xxx.yy"
  netif: '{"net0":"name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0"}'
  os_template_name: "debian-12-standard_12.2-1_amd64.tar.zst"

  vmid: 180
  node: pve1
  hostname: "k8s-{{ vmid }}"
  cpu_cores: 2
  memory: 1024
  rootfs_disk_size: "8"
  storage: "Ceph_1TB"

  module_defaults:
  ansible.builtin.get_url:
  validate_certs: false
  tasks:

  • name: Install Proxmoxer"
    ansible.builtin.apt:
    name: python3-proxmoxer
    state: latest
    update_cache: true

  • name: Check if LXC template is downloaded
    ansible.builtin.stat:
    path: "/var/lib/vz/template/cache/{{ os_template_name }}"
    register: p

  • ansible.builtin.debug: msg="Template was already downloaded. Skipping."
    when: p.stat.exists

  • name: Show IP
    ansible.builtin.shell: ip a

  • name: Download LXC template
    ansible.builtin.shell: pveam download local {{ os_template_name }}
    when: p.stat.exists == False

  • name: Create K8S LXC-Container
    connection: local
    community.general.proxmox:
    vmid: "{{ vmid }}"
    node: "{{ node }}"
    state: "present"

    purge: true

    onboot: true
    api_user: "{{ username }}"
    api_password: "{{ user_password }}"
    api_host: "{{ api_host }}"
    password: "{{ lxc_root_password }}"
    hostname: "{{ hostname }}-pve1"
    ostemplate: "local:vztmpl/{{ os_template_name }}"
    storage: "{{ storage }}"
    netif: "{{ netif }}"
    memory: "{{ memory }}"
    disk: "{{ rootfs_disk_size }}"
    swap: "0"
    unprivileged: false
    features:

    • fuse=1
    • nesting=1
      pubkey: ""
      register: container_present
      tags:
    • create

  • name: Check if K8S LXC is running
    ansible.builtin.shell: pct status {{ vmid }}
    register: lxc_running
    tags:

    • create
    • restart
    • lxc_started

  • name: Insert lines into LXC conf
    ansible.builtin.blockinfile:
    path: "/etc/pve/lxc/{{ vmid }}.conf"
    block: |
    lxc.apparmor.profile: unconfined
    lxc.cap.drop:
    lxc.mount.auto: proc:rw sys:rw
    lxc.mount.entry: /sys/kernel/security sys/kernel/security none bind,create=file 0 0
    backup: yes
    when: '"status: running" not in lxc_running.stdout'
    tags:

    • lxc_started
    • create
    • restart

  • name: Start K8S LXC container
    community.general.proxmox:
    vmid: "{{ vmid }}"
    node: "{{ node }}"
    state: "started"
    api_user: "{{ username }}"
    api_password: "{{ user_password }}"
    api_host: "{{ api_host }}"
    when: '"status: running" not in lxc_running.stdout'
    tags:

    • lxc_started
    • create
    • restart

  • name: get K8S LXC Container IP
    ansible.builtin.shell:
    cmd: pct exec {{ vmid }} ip addr show|grep 'inet 10'| awk -F ' ' '{print $2}'
    register: lxcip

  • name: show K8S LXC Container IP
    ansible.builtin.debug:
    msg: The Container {{ vmid }} has the following IP {{ lxcip.stdout }}
    when: lxcip.stdout != "" %`

`- hosts: "{{ machines }}"
#remote_user: "{{ remote_user }}"
tasks:

• name: Add CronJob "@reboot ln -s /dev/console /dev/kmsg"
  ansible.builtin.cron:
  name: "create /dev/kmsg"
  special_time: reboot
  job: "ln -s /dev/console /dev/kmsg"

• name: Install required Packages
  apt:
  pkg:

  • snapd
  • squashfuse
  • fuse
  • sudo
    state: latest
    update_cache: true

• name: Reboot
  ansible.builtin.reboot:
  msg: "Reboot initiated by Ansible"
  connect_timeout: 5
  reboot_timeout: 600
  pre_reboot_delay: 0
  post_reboot_delay: 30
  test_command: whoami

• name: Install microk8s
  community.general.snap:
  name: microk8s
  state: absent
  classic: true`

'- hosts: all

vars:
ansible_ssh_private_key_file: ~/.ssh/key
ansible_ssh_user: root

tasks:

• name: Write the LXCs host key to known_hosts
  connection: local
  shell: "ssh-keyscan -H {{ inventory_hostname }} >> ~/.ssh/known_hosts"

• name: Add CronJob "@reboot ln -s /dev/console /dev/kmsg"
  ansible.builtin.cron:
  name: "create /dev/kmsg"
  special_time: reboot
  job: "ln -s /dev/console /dev/kmsg"

• name: Install required Packages
  apt:
  pkg:

  • snapd
  • squashfuse
  • fuse
  • sudo
    state: latest
    update_cache: true

• name: Reboot
  ansible.builtin.reboot:
  msg: "Reboot initiated by Ansible"
  connect_timeout: 5
  reboot_timeout: 30
  pre_reboot_delay: 0
  post_reboot_delay: 10
  test_command: whoami

• name: Install microk8s
  community.general.snap:
  name: microk8s
  state: present
  classic: true'

Run them in the listed order and you'll have your 1st microk8s LXC

After two days of trying EVERYTHING, this is how i made it work on Proxmox 8.0.3:

1- Download debian 12 (debian-12-standard_12.2-1_amd64.tar.zst) ct template ( i don't know why, but i couldn't make it work with ubuntu ):

pveam download local debian-12-standard_12.2-1_amd64.tar.zst

2- Make a new CT with the following configurations:

• Make sure "Unprivileged Container" is unchecked, so it becomes privileged.
• Use downloaded debian template.
• Make sure sure swap is 0 in memory tab.
  NOTE: Don't start the CT yet.

3- On the created CT go to options tab, double click on "Features" and enable "Fuse" and "Nesting".
4- SSH into your Proxmox node and navigate to /etc/pve/lxc/, and open your {ct_id}.conf and add these few lines in:

lxc.apparmor.profile: unconfined
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
lxc.mount.entry: /sys/kernel/security sys/kernel/security none bind,create=file 0 0

5- Start the CT
6- Open your crontab using crontab -e and add this line at the end:

@reboot ln -s /dev/console /dev/kmsg

7- Install required apt packages:

apt update && apt upgrade -y && apt install snapd squashfuse fuse sudo -y

8- Reboot for changes to take effect.
9- After the reboot, finally you can install microk8s:

snap install microk8s --classic

10- Enjoy your first microk8s on Proxmox 🎉

perchance someone knows why I can't SSH in? "Permission denied, please try again." Everything else seems to work...

@stealthmachines, did you add your ssh keys or specified password when creating a container?

snap install microk8s --classic

Your original suggestion did not work for me in both debian and ubuntu but was a really good starting place.
The following conf works in both ubuntu and debian

root@ns26031989:~# cat /etc/pve/lxc/107.conf 
arch: amd64
cores: 2
features: mount=fuse,nesting=1
hostname: runUp
memory: 4096
net0: name=eth0,bridge=vmbr0,firewall=1,gw=149.202.93.254,hwaddr=02:00:00:3a:88:6e,ip=178.32.45.21/24,type=veth
ostype: debian
rootfs: local:107/vm-107-disk-0.raw,size=80G
swap: 0
lxc.apparmor.profile: unconfined
lxc.cap.drop: 
lxc.mount.auto: proc:rw sys:rw
lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file 0 0
lxc.mount.entry: /sys/kernel/security sys/kernel/security none bind,create=file 0 0

Excellent work, guys. You saved me hours....Thank you so much. I got one addtion to make, though:
It is important to assign static ip adresses to the container because due to some conflict in the configuration, dhcp does not seem to work anymore.