This is a writedown of how I installed my raspberry pi(s).
The base is a Proxmox, so I don't have to worry about reformat the SD card every time I wanna try out something new, as I can start VMs and LXC containers, as well as Docker containers once it's configured properly.
Note: Those files are versioned, so you can always look what changed from time to time.
Run
sudo raspi-config8 Update Update this tool to the latest version
5 Localisation Options Configure language and regional settings
L1 Locale Configure language and regional settings[ ] the en_GB.UTF-8 UTF-8 or any other locale with the Spacebar key.[*] the en_US.UTF-8 UTF-8 with the Spacebar key.en_US.UTF-8Flash and startup the latest image from https://downloads.raspberrypi.org/raspios_arm64/.
Raspberry Pi OS (other) from the listRaspberry Pi OS Lite (64-bit) from the listpimox, add your ssh key as well as set up wifi.Set up your ssh key if you didn't with the flashing tool and disable password login. If not at least change the default password.
sudo -s
curl https://raw.githubusercontent.com/pimox/pimox7/master/RPiOS64-IA-Install.sh > RPiOS64-IA-Install.sh
chmod +x RPiOS64-IA-Install.sh
./RPiOS64-IA-Install.sh
Follow the prompts
pimox192.168.178.30/24y#########################################################################################
=========================================================================================
THE NEW HOSTNAME WILL BE: pimox-wg
=========================================================================================
THE DHCP SERVER ( dhcpcd5 ) WILL BE REMOVED !!!
=========================================================================================
THE PIMOX REPO WILL BE ADDED IN : /etc/apt/sources.list.d/pimox.list CONFIGURATION :
# Pimox 7 Development Repo
deb https://raw.githubusercontent.com/pimox/pimox7/master/ dev/
=========================================================================================
THE NETWORK CONFIGURATION IN : /etc/network/interfaces WILL BE CHANGED !!! TO :
auto lo
iface lo inet loopback
iface eth0 inet manual
auto vmbr0
iface vmbr0 inet static
address 192.168.178.30/24
gateway 192.168.178.1
bridge-ports eth0
bridge-stp off
bridge-fd 0
=========================================================================================
THE HOSTNAMES IN : /etc/hosts WILL BE OVERWRITTEN !!! WITH :
127.0.0.1 localhost
192.168.178.30 pimox-wg
=========================================================================================
THESE STATEMENTS WILL BE ADDED TO THE /boot/cmdline.txt IF NONE EXISTENT :
cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1
=========================================================================================
#########################################################################################
y=========================================================================================
! SETUP NEW ROOT PASSWORD !
=========================================================================================
••••••••••••••••••••••Work around the issues:
a) Apply ToSMaverick's header install trick, downgrading the kernel:
wget http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/raspberrypi-kernel_1.20220120-1_arm64.deb wget http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/raspberrypi-kernel-headers_1.20220120-1_arm64.deb sudo dpkg -i raspberrypi-kernel_1.20220120-1_arm64.deb raspberrypi-kernel-headers_1.20220120-1_arm64.deb sudo apt-mark hold raspberrypi-kernel raspberrypi-kernel-headers
b) Apply rgsilva's compile trick, and install ceph-dkms from scratch
ceph-dkms
sudo -i
apt install git debhelper (ignore the errors, we are working on resolving those right now.)
First you need to clone RPi's Kernel source code.
git clone https://github.com/raspberrypi/linux.git --depth=1
Then copy all ceph files into a new default-src directory.
Note that the third line needs to copy to default-src/… not sdefault-srcrc/…. This is corrected below.
mkdir -p default-src/drivers/block # typo cp -r linux/drivers/block/{rbd.c,rbd_types.h} default-src/drivers/block/ grep -E "^#|^ccflags|rbd.o" linux/drivers/block/Makefile > default-src/drivers/block/Makefile mkdir -p default-src/fs/ceph cp linux/fs/ceph/* default-src/fs/ceph/ mkdir -p default-src/net/ceph cp -r linux/net/ceph/* default-src/net/ceph/
Now you need to clone the original repository
git clone https://github.com/pimox/ceph-dkms.git
Then you need to replace the default-src directory with the one you created in the step before
rm -rf ceph-dkms/src/default-src cp -r default-src ceph-dkms/src/default-src
To make sure the deb won't get (easily) replaced, we can increase its minor version. This step is optional and can cause issues with future updates, but I recommend it.
Maybe it's better not do this and instead pin the version at the end. (That's what I'm attempting)
( cd ceph-dkms/ sed -i "s/0\\.0\\.2/0.0.2-1/g" debian/changelog sed -i "s/0\\.0\\.2/0.0.2-1/g" debian/dkms )
Now you need to regenerate the .deb
(cd ceph-dkms && make)
You can finally install it using dpkg. Note: if you didn't change the version before, you need to update the package version below.
Because I didn't modify the version, instead of
( cd ceph-dkms sudo dpkg -i ceph-dkms_0.0.2-1_all.deb )
I did run
(
cd ceph-dkms
sudo dpkg -i ceph-dkms_0.0.2_all.deb
)
Make sure it's locked to not update
sudo apt-mark hold ceph-dkms
zfs-dkms
0. For the zfs-dkms, you can just follow OpenZFS's instructions for installing on Debian Bullseye.
echo 'deb http://deb.debian.org/debian bullseye-backports main contrib' > /etc/apt/sources.list.d/bullseye-backports.list
echo 'deb-src http://deb.debian.org/debian bullseye-backports main contrib' >> /etc/apt/sources.list.d/bullseye-backports.list
echo 'Package: libnvpair1linux libnvpair3linux libuutil1linux libuutil3linux libzfs2linux libzfs4linux libzpool2linux libzpool4linux spl-dkms zfs-dkms zfs-test zfsutils-linux zfsutils-linux-dev zfs-zed' > /etc/apt/preferences.d/90_zfs
echo 'Pin: release n=bullseye-backports' >> /etc/apt/preferences.d/90_zfs
echo 'Pin-Priority: 990' >> /etc/apt/preferences.d/90_zfs
apt update
apt install dpkg-dev
linux-headers-$(uname -r) because of Couldn't find any package by glob 'linux-headers-5.15.32-v8'linux-image-amd64 as that would be the wrong achitecture, and linux-image-aarch64 was not available.DEBIAN_FRONTEND=noninteractive apt install zfs-dkms zfsutils-linux
ceph-dkms and zfs-dkms packages exist.
>
> As of today, at least, this seems to be fully working. Just tested on both nodes on my cluster and they all seem pretty happy with the new packages:
>
> > $ apt list --installed | grep -E "(ceph|zfs)-dkms" > > ceph-dkms/now 0.0.2-1 all [installed,local] > zfs-dkms/bullseye-backports,bullseye-backports,now 2.1.4-1~bpo11+1 all [installed,automatic] >
>generic-aarch64generic-aarch64 the image from the home assistant release page
wget https://github.com/home-assistant/operating-system/releases/download/8.0.rc4/haos_generic-aarch64-8.0.rc4.img.xz
xz --decompress haos_generic-aarch64-*.img.xz
mv haos_generic-aarch64-8.0.rc4.img /var/lib/vz/template/iso/haos_generic-aarch64-8.0.rc4.img
# this needs sudo. It did work in the shell on the website.
qm create 100 --bios ovmf --cores 2 --memory 4096 --scsihw virtio-scsi-pci -net0 virtio,bridge=vmbr0
# this needs sudo. It did work in the shell on the website.
qm importdisk 100 /var/lib/vz/template/iso/haos_generic-aarch64-*.img local
Successfully imported disk as 'unused0:local:100/vm-100-disk-0.raw'
Hardware TabEditBus/Device drop down select SATA.Add button.Options Tab:Name: home-assistantStart at Boot: Check yes.Boot Order: Set the boot disk to the newly added one.Starthttp://homeassistant.local:8123)Following pimox/pimox7#48 (comment)
Device column, in this case that's 10c4:ea60.
Port column, in this case that's 1-1.3.Device or Port) when you selected your device.
10c4:ea60 for the Device ID, or 1-1.3 if you went for the Port.
lsusb without your stick pluggend itBus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
lsusb with your stick pluggend itBus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 10c4:ea60 Silicon Labs CP210x UART Bridge
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
The newly added entry is your device:
Bus 001 Device 003: ID 10c4:ea60 Silicon Labs CP210x UART BridgeThose lines have the format Bus AAA Device BBB: ID CCCC:DDDD blahblah, so
CCCC:DDDD, in this case 10c4:ea60.A-A.B, in this case 1-1.3.
100./etc/pve/qemu-server/<VM number>.confsudo pico /etc/pve/qemu-server/100.confargs:
-device qemu-xhci, to have the USB drives for the remote console (fake mouse and fake keyboard)AAAA:BBBB: Add -device usb-host,vendorid=0x<AAAA>,productid=0x<BBBB>
10c4:ea60 it would be -device usb-host,vendorid=0x10c4,productid=0xea60.A-A.B: Add -device usb-host,hostbus=<A>,hostport=<A.B>
1-1.3 it would be -device usb-host,hostbus=1,hostport=1.3.args: -device qemu-xhci -device usb-host,vendorid=<0xVendorid>,productid=<0xProductid>In my case for the SONOFF Zigbee stick:
args: -device qemu-xhci -device usb-host,vendorid=0x10c4,productid=0xea60For having the same stick two times in that (i.e. same vendor + product id) you have to use the port number.
args: -device qemu-xhci -device usb-host,hostbus=1,hostport=1.2 -device usb-host,hostbus=1,hostport=1.3
First you need an 64bit ARM template.
This is also known as Root FS. Basically a bunch of files that make the chosen linux flavor special.
If you are coming from the Docker world, basically that's the FROM debian step.
For clarity sake, even if the numerical IDs for CTs share the starting point of 100 with the VMs, I will use 100-199 for VMs exclusively, and 200-299 for LXC containers. Should I ever breach that limit, I will always add blocks in an alternatig pattern, i.e. it is VMs if the hundreds place is even, and CTs if it's odd.
This also means my first CT will have the id 200 in this guide.
Now we are ready to go
Raspberry pi still uses arm, but thanks to the Pi 4B you can use the 64 bit version, hence arm64.
Choose your linux flavor
Find a rootfs archive file
rootfs.zip, rootfs.tar, rootfs.tar.gz, rootfs.tar.xz or similar.rootfs.tar.xz in /images/debian/bullseye/arm64/default/20220505_06:03.
Download the rootfs archive,
…and give it a proper name:
You need to modify the $DATE variable with an existing date from the url above.
Note, if you picked ubuntu, your $ROOTFS_DOWNLOAD_URL needs to be adapted as well.
DATE="20220616_05:25"
TEMPLATE_FILE_NAME="debian-bullseye-arm64_${DATE}_rootfs.tar.xz"
ROOTFS_DOWNLOAD_URL="https://uk.lxd.images.canonical.com/images/debian/bullseye/arm64/default/${DATE}/rootfs.tar.xz"
wget "${ROOTFS_DOWNLOAD_URL}" -O "${TEMPLATE_FILE_NAME}"If the server serves them on different paths in the future you need to adopt $ROOTFS_DOWNLOAD_URL as needed.
Move it to the CT (Container Template) directory
mv ${TEMPLATE_FILE_NAME} /var/lib/vz/template/cache/${TEMPLATE_FILE_NAME}Confirm it's existance in the GUI
Datacenter, your node (e.g. pimox), local, then in the middle CT Images.
Crate an CT, either gui or via terminal
pct create 200 /var/lib/vz/template/cache/${TEMPLATE_FILE_NAME} --arch arm64 --features nesting=1 --hostname docker --ostype debian --net0 name=eth0,bridge=vmbr0,firewall=1,ip=dhcp,ip6=dhcp --password='much $ecure passw0rd' --tags docker-hostpct create The command to create a CT container200 : (1 - N) The (unique) ID of the VM. See About VM Ids./var/lib/vz/template/cache/${TEMPLATE_FILE_NAME} The OS template or backup file.--arch arm64 Architecture for the Pi.--features nesting=1 for docker.--hostname docker hostname and how it will be called in the local network.--ostype debian That's what we choose above.--net0 name=eth0,bridge=vmbr0,firewall=1,ip=dhcp,ip6=dhcp--ssh-public-keys ~/.ssh/id_rsa.pub the root user of the Pimox/Proxmox install.--ssh-public-keys ~/.ssh/remote_id_rsa.pub my own one, for my laptop to connect.--password='much $ecure passw0rd' as alternative to the ssh-public-keys command above, which seems to be failing unfortunantely.--tags docker-host Tags of the Container. This is only meta information.Note, most of those options can be edited/added with the CLI later as well, e.g.:
pct set 200 -net0 name=eth0,bridge=vmbr0,firewall=1,ip=dhcp,ip6=dhcpThe net0 one seems to fail with unable to open file '/etc/network/interfaces.tmp.205501' - No such file or directory
pct shutdown 200
// echo 'lxc.apparmor.profile: lxc-default-with-nesting' | sudo tee -a /etc/pve/lxc/200.conf
pct start 200
pct pull 200 /root/.ssh/authorized_keys authorized_keys.200.bak If there's no such file, that's okey.
pct push 200 /root/.ssh/authorized_keys /root/.ssh/authorized_keys Copy our authorized file to the CT.
Verbose start for debugging: lxc-start -n 200 -F -l DEBUG -o /tmp/lxc-200.log
Here is a list of links I looked at.
- https://danthesalmon.com/running-docker-on-proxmox/
- https://kevingoos.medium.com/kubernetes-inside-proxmox-lxc-cce5c9927942
- pimox/pimox7#14 (comment)
- pimox/pimox7#14 (comment)
- https://pve.proxmox.com/pve-docs/pct.1.html
- https://gist.github.com/luckydonald/1849291fb5e19c87df8c8a1618e29eaa?permalink_comment_id=4601623#gistcomment-4601623
wget https://cdimage.debian.org/debian-cd/11.4.0/arm64/iso-dvd/debian-11.4.0-arm64-DVD-1.isomv debian-11.4.0-arm64-DVD-1.iso /var/lib/vz/template/iso/debian-11.4.0-arm64-DVD-1.iso
@nistvan86 I don't really have good experience or benchmarks because I only tried this route and never any other. HA isn't very demanding I think (i don't use cameras etc, just simple zigbee), so it doesn't matter much.
With the snapshot functionality of proxmox NOT working I however don't see a big Argument for proxmox if you only run HA - like I do
@luckydonald thanks!
I too had the "unable to open file '/etc/network/interfaces.tmp.xxxx' - No such file or directory" issue.
I've just figured out how to get the latest Cloud image of Debian 12 Bullseye working on PiMox 7:
https://uk.lxd.images.canonical.com/images/debian/bookworm/arm64/cloud/20230614_05:24/rootfs.tar.xz
When configuring the CT, leave IP = Static. Do not enter any IP addresses anywhere.
Boot the CT, enter the CTs console.
Install Nano (you can use VI/VIM, but I'm not a fan of them)
apt install nano
Edit the network configuration file
nano /etc/systemd/network/eth0.network
Change:
[Network]
DHCP=yes
to
[Network]
Address=192.168.1.101/24
Gateway=192.168.1.1
Restart Systemd Networkd
systemctl restart systemd-networkd
Check the static IP is set (you'll probably also see the DHCP IP is still there)
ip -c a
Reboot
reboot now
You've now got the static IP set up
@AubsUK thanks! I linked to that at that section above.
I too had the "unable to open file '/etc/network/interfaces.tmp.xxxx' - No such file or directory" issue.
I've just figured out how to get the latest Cloud image of Debian 12 Bullseye working on PiMox 7:
https://uk.lxd.images.canonical.com/images/debian/bookworm/arm64/cloud/20230614_05:24/rootfs.tar.xz
When configuring the CT, leave IP = Static. Do not enter any IP addresses anywhere.
Boot the CT, enter the CTs console.
Install Nano (you can use VI/VIM, but I'm not a fan of them)
apt install nanoEdit the network configuration file
nano /etc/systemd/network/eth0.networkChange:
[Network]
DHCP=yesto
[Network]
Address=192.168.1.101/24
Gateway=192.168.1.1Restart Systemd Networkd
systemctl restart systemd-networkdCheck the static IP is set (you'll probably also see the DHCP IP is still there)
ip -c aReboot
reboot nowYou've now got the static IP set up
Why I lost the static IP after I reboot my container? The static IP seems not be saved. The static IP comes again after run
systemctl restart systemd-networkd
Maybe a bit late to the party...
Yes for Debian CT's you need to setup your network manually within the CT and yes you lose your networkconnectivety after a reboot.
That's because systemd-networkd isn't started automatically.
There are 2 options.
Option 1: Run systemctl enable systemd-networkd to have network after a reboot.
Option 2: Make sure you have networkconnectivety and then run apt install ifupdown2.
After that you'll be able to setup your network in the CT properties as you can with Ubuntu CT's.
There doesn't seem to be too much info about this on the net so forgive me if this is the wrong place to ask (and delete if you like).
How does the performance compare eg. to a native Raspbian + Docker setup when running containers on top of Pimox (which I think uses KVM) + Debian + Docker? Is it usable? I'm thinking about migrating my current Raspbian + dockerized HA to a Pimox + HAOS + Debian/Docker (for rest of the containers) setup, but I'm unsure if it's a wise idea to do it or not.