Skip to content

Instantly share code, notes, and snippets.

@devimc

devimc/hotplug.md

Last active October 14, 2024 08:31
Show Gist options
  • Save devimc/e9fd533e52b08387f1df65df8b19e038 to your computer and use it in GitHub Desktop.
Save devimc/e9fd533e52b08387f1df65df8b19e038 to your computer and use it in GitHub Desktop.
Download ZIP
Hot plugging devices
Raw
hotplug.md

Hot plugging devices using ACPI, SHPC and native

This gist is to show different ways to hot plug devices in a Virtual Machine

Set up the environment

Create a device mapper

dd if=/dev/zero of=devmap.img count=1 bs=50M
fdisk devmap.img
	# Command (m for help): g
	# Command (m for help): n
	# Partition number (1-128, default 1): <RETURN>
	# First sector (2048-102366, default 2048): <RETURN>
	# Last sector, +sectors or +size{K,M,G,T,P} (2048-102366, default 102366): <RETURN>
	# Command (m for help): w
fdisk -l devmap.img
	# Disk devmap.img: 50 MiB, 52428800 bytes, 102400 sectors
sudo partx -av devmap.img
	# /dev/loop0: partition #1 added
sudo mkfs.ext4 /dev/loop0p1
sudo dmsetup create devmap --table '0 102400 linear /dev/loop0 0'

Do not forget remove it once you finish with your tests

ls -l /dev/mapper/devmap
	#  /dev/mapper/devmap -> ../dm-4
sudo dmsetup remove -f /dev/dm-4
sudo partx -d /dev/loop4

Setup environment variables

export KERNEL_CMDLINE="root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 \
tsc=reliable no_timer_check rcupdate.rcu_expedited=1 pci=lastbus=0 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 \
i8042.noaux=1 noreplace-smp noacpi noapic reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off \
cryptomgr.notests net.ifnames=0 console=ttyS0"
export Q35_ACCEL="kvm,kernel_irqchip,nvdimm,nosmm,nosmbus,nosata,nopit,static-prt,nofw"

Get and build tools

clear containers qemu 2.9

compared with qemu upstream this version of qemu offers us two extra machine accelerators static-prt and nofw

git clone https://github.com/clearcontainers/qemu ;
git checkout qemu-lite-v2.9.0 ;
pushd qemu ;
curl https://raw.githubusercontent.com/clearcontainers/packaging/master/qemu-lite/configure.patch | patch -p1 ;
./configure --disable-static --disable-bluez --disable-brlapi --disable-bzip2 --disable-curl \
--disable-curses --disable-debug-tcg --disable-fdt --disable-glusterfs --disable-gtk \
--disable-libiscsi --disable-libnfs --disable-libssh2 --disable-libusb --disable-linux-aio \
--disable-lzo --disable-opengl --disable-qom-cast-debug --disable-rbd --disable-rdma --disable-sdl \
--disable-seccomp --disable-slirp --disable-snappy --disable-spice --disable-strip \
--disable-tcg-interpreter --disable-tcmalloc --disable-tools --disable-tpm --disable-usb-redir \
--disable-uuid --disable-vnc --disable-vnc-jpeg --disable-vnc-png --disable-vnc-sasl --disable-vte \
--disable-xen --enable-attr --enable-cap-ng --enable-kvm --enable-virtfs \
--target-list=x86_64-softmmu --extra-cflags="-fno-semantic-interposition -O3 -falign-functions=32" \
--datadir=/usr/share/qemu-lite --libdir=/usr/lib64/qemu-lite --libexecdir=/usr/libexec/qemu-lite \
--enable-vhost-net --disable-docs ;
cp x86_64-softmmu/qemu-system-x86_64 ../qemu-lite-system-x86_64-2.9.0
popd

linux container kernel

Patches (Only for SHPC)

Configs

for SHPC

  • CONFIG_HOTPLUG_PCI_PCIE=y
  • CONFIG_HOTPLUG_PCI=y
  • CONFIG_HOTPLUG_PCI_CPCI=y
  • CONFIG_HOTPLUG_PCI_SHPC=y

for native hotplug

  • CONFIG_HOTPLUG_PCI_PCIE=y
  • CONFIG_PCIEAER=y
  • CONFIG_PCIEASPM=y
  • CONFIG_PCIEASPM_DEFAULT=y
  • CONFIG_HOTPLUG_PCI=y
  • CONFIG_RAS=y

qboot

git clone https://github.com/bonzini/qboot
make

CBFS linux kernel

dd if=/dev/zero of=boot.bin bs=4096 count=1
cbfstool cbfs.rom create -s 8128k -B boot.bin -m x86 -o 0x1000
cbfstool cbfs.rom add -f vmlinuz-4.9.47-77.container -n vmlinuz -t raw
echo ${KERNEL_CMDLINE} > cmdline
cbfstool cbfs.rom add -f cmdline -n cmdline -t raw

Native hot plug

q35 + vmlinux + nofw

Start a new VM

./qemu-lite-system-x86_64-2.9.0 -machine q35,accel=${Q35_ACCEL} -cpu host -device \
nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=containers.img,size=471859200 \
-m 2048M,slots=2,maxmem=16998M -smp 4,cores=4,threads=1,sockets=1 -vga none -nographic \
-no-user-config -nodefaults -kernel vmlinux.container -append "${KERNEL_CMDLINE}" -device \
virtio-serial-pci,id=virtio-serial0 -chardev stdio,id=charconsole0,signal=off -device \
virtconsole,chardev=charconsole0,id=console0 -enable-kvm -qmp unix:hypervisor.sock,server,nowait \
-device pcie-root-port,id=root1

Hot plugging virtio disk

echo '{ "execute": "qmp_capabilities" }
{ "execute": "blockdev-add", "arguments":{"driver":"raw","node-name":"disk1", "file":{"driver":"file","filename":"/dev/mapper/devmap"}}}
{ "execute": "device_add", "arguments": { "driver": "virtio-blk", "drive": "disk1", "id": "virtio0", "bus": "root1", "bootindex": "1" } }' |
socat - unix-connect:hypervisor.sock

log message example

[   11.391260] pciehp 0000:00:02.0:pcie004: Slot(0): Attention button pressed
[   11.391382] pciehp 0000:00:02.0:pcie004: Slot(0): Card present
[   11.391512] pciehp 0000:00:02.0:pcie004: Slot(0) Powering on due to button press
[   12.516403] pci 0000:01:00.0: BAR 4: assigned [mem 0xf00000000-0xf00003fff 64bit pref]
[   12.516712] pci 0000:01:00.0: BAR 1: assigned [mem 0xc0000000-0xc0000fff]
[   12.516874] pcieport 0000:00:02.0: PCI bridge to [bus 01]
[   12.517008] pcieport 0000:00:02.0:   bridge window [io  0x1000-0x1fff]
[   12.518776] pcieport 0000:00:02.0:   bridge window [mem 0xc0000000-0xc01fffff]
[   12.520490] pcieport 0000:00:02.0:   bridge window [mem 0xf00000000-0xf001fffff 64bit pref]
[   12.523949] virtio-pci 0000:01:00.0: enabling device (0000 -> 0002)
[   12.527777]  vda: vda1

Hot plugging virtio net

echo '{ "execute": "qmp_capabilities" }
{ "execute": "device_add", "arguments": { "driver": "virtio-net-pci", "id": "virtio-net1", "bus": "root1" } }' |
socat - unix-connect:hypervisor.sock

log message example

[   11.827202] pciehp 0000:00:02.0:pcie004: Slot(0): Attention button pressed
[   11.827342] pciehp 0000:00:02.0:pcie004: Slot(0): Card present
[   11.827474] pciehp 0000:00:02.0:pcie004: Slot(0) Powering on due to button press
[   12.962931] pci 0000:01:00.0: BAR 4: assigned [mem 0xf00000000-0xf00003fff 64bit pref]
[   12.963436] pci 0000:01:00.0: BAR 1: assigned [mem 0xc0000000-0xc0000fff]
[   12.963704] pcieport 0000:00:02.0: PCI bridge to [bus 01]
[   12.963899] pcieport 0000:00:02.0:   bridge window [io  0x1000-0x1fff]
[   12.966579] pcieport 0000:00:02.0:   bridge window [mem 0xc0000000-0xc01fffff]
[   12.968503] pcieport 0000:00:02.0:   bridge window [mem 0xf00000000-0xf001fffff 64bit pref]
[   12.970227] virtio-pci 0000:01:00.0: enabling device (0000 -> 0002)

Hot plugging virtio serial port

To hot plug a virtio serial port PCI is not needed

echo '{ "execute": "qmp_capabilities" }
{ "execute" : "chardev-add", "arguments" : { "id" : "charch1", "backend" : { "type" : "socket", "data" : {"wait": false, "server": true, "addr": { "type": "unix", "data": { "path": "tty.sock" } } } } } }
{ "execute": "device_add", "arguments": { "driver": "virtserialport", "chardev":"charch1", "id": "channel1", "name": "sh.hyper.channel.1" } }' |
socat - unix-connect:hypervisor.sock

Native hot plug (PCIe switch)

q35 + vmlinux + nofw

Start a new VM

./qemu-lite-system-x86_64-2.9.0 -machine q35,accel=${Q35_ACCEL} -cpu host -device \
nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=containers.img,size=471859200 \
-m 2048M,slots=2,maxmem=16998M -smp 4,cores=4,threads=1,sockets=1 -vga none -nographic \
-no-user-config -nodefaults -kernel vmlinux.container -append "${KERNEL_CMDLINE}" -device \
virtio-serial-pci,id=virtio-serial0 -chardev stdio,id=charconsole0,signal=off -device \
virtconsole,chardev=charconsole0,id=console0 -enable-kvm -qmp unix:hypervisor.sock,server,nowait \
-device ioh3420,id=root1,chassis=1,slot=1

Hot plugging virtio disk

echo '{ "execute": "qmp_capabilities" }
{ "execute": "device_add", "arguments": { "driver": "xio3130-downstream", "id": "downstream1", "bus": "root1", "chassis": "0", "slot": "0" } }' |
socat - unix-connect:hypervisor.sock

echo '{ "execute": "qmp_capabilities" }
{ "execute": "blockdev-add", "arguments":{"driver":"raw","node-name":"disk1", "file":{"driver":"file","filename":"/dev/mapper/devmap" } } }
{ "execute": "device_add", "arguments": { "driver": "virtio-blk", "drive": "disk1", "id": "virtio1", "bus": "downstream1", "bootindex": "0" } }' |
socat - unix-connect:hypervisor.sock

ACPI hot plug

pc + vmlinuz

Start a new VM

./qemu-lite-system-x86_64-2.9.0 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -device \
nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=clear-containers.img,size=471859200 \
-m 2048M,slots=2,maxmem=16998M -smp 4,cores=4,threads=1,sockets=1 -vga none -nographic \
-no-user-config -nodefaults -kernel vmlinuz.container -append "${KERNEL_CMDLINE}" -device \
virtio-serial-pci,id=virtio-serial0 -chardev stdio,id=charconsole0,signal=off -device \
virtconsole,chardev=charconsole0,id=console0 -enable-kvm -qmp unix:hypervisor.sock,server,nowait

Hot plugging virtio disk

echo '{ "execute": "qmp_capabilities" }
{ "execute": "blockdev-add", "arguments":{"driver":"raw","node-name":"disk1", "file":{"driver":"file","filename":"/dev/mapper/devmap" } } }
{ "execute": "device_add", "arguments": { "driver": "virtio-blk", "drive": "disk1", "id": "virtio1", "bus": "pci.0", "bootindex": "1" } }' |
socat - unix-connect:hypervisor.sock

to detect the block device the pci bus MUST be re-scanned inside the VM

echo 1 > /sys/bus/pci/rescan

log message example

[   34.328712] pci 0000:00:04.0: BAR 4: assigned [mem 0x80000000-0x80003fff 64bit pref]
[   34.329176] pci 0000:00:04.0: BAR 1: assigned [mem 0x80004000-0x80004fff]
[   34.329612] pci 0000:00:04.0: BAR 0: assigned [io  0x1000-0x103f]
[   34.329947] pci 0000:00:03.0: PCI bridge to [bus 01]
[   34.332923] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)
[   34.350844] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
[   34.353554]  vda: vda1

ACPI hot plug (pci-bridge)

pc + vmlinuz

Start a new VM

./qemu-lite-system-x86_64-2.9.0 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -device \
nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=clear-containers.img,size=471859200 \
-m 2048M,slots=2,maxmem=16998M -smp 4,cores=4,threads=1,sockets=1 -vga none -nographic \
-no-user-config -nodefaults -kernel vmlinuz.container -append "${KERNEL_CMDLINE}" -device \
virtio-serial-pci,id=virtio-serial0 -chardev stdio,id=charconsole0,signal=off -device \
virtconsole,chardev=charconsole0,id=console0 -enable-kvm -qmp unix:hypervisor.sock,server,nowait \
-device pci-bridge,id=root1,chassis_nr=1

Hot plugging virtio disk

echo '{ "execute": "qmp_capabilities" }
{ "execute": "blockdev-add", "arguments":{"driver":"raw","node-name":"disk1", "file":{"driver":"file","filename":"/dev/mapper/devmap" } } }
{ "execute": "device_add", "arguments": { "driver": "virtio-blk", "drive": "disk1", "id": "virtio1", "bus": "root1", "bootindex": "1" } }' |
socat - unix-connect:hypervisor.sock

to detect the block device the pci bus MUST be re-scanned inside the VM

echo 1 > /sys/bus/pci/rescan

log message example

[   28.189942] pci 0000:00:03.0: BAR 8: assigned [mem 0x80000000-0x800fffff]
[   28.190097] pci 0000:00:03.0: BAR 9: assigned [mem 0x80100000-0x801fffff 64bit pref]
[   28.190202] pci 0000:00:03.0: BAR 7: assigned [io  0x1000-0x1fff]
[   28.190287] pci 0000:01:00.0: BAR 4: assigned [mem 0x80100000-0x80103fff 64bit pref]
[   28.190505] pci 0000:01:00.0: BAR 1: assigned [mem 0x80000000-0x80000fff]
[   28.190651] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x103f]
[   28.190785] pci 0000:00:03.0: PCI bridge to [bus 01]
[   28.190892] pci 0000:00:03.0:   bridge window [io  0x1000-0x1fff]
[   28.192473] pci 0000:00:03.0:   bridge window [mem 0x80000000-0x800fffff]
[   28.193584] pci 0000:00:03.0:   bridge window [mem 0x80100000-0x801fffff 64bit pref]
[   28.197029] virtio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[   28.215856] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11
[   28.219026]  vda: vda1

ACPI hot plug several devices

q35 + nofw + vmlinux

Start a new VM

./qemu-lite-system-x86_64-2.9.0 -machine q35,accel=$Q35_ACCEL -cpu host -device \
nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=clear-containers.img,size=471859200 \
-m 2048M,slots=2,maxmem=16998M -smp 4,cores=4,threads=1,sockets=1 -vga none -nographic \
-no-user-config -nodefaults -kernel vmlinux.container -append "$KERNEL_CMDLINE" \
-device virtio-serial-pci,id=virtio-serial0 -chardev stdio,id=charconsole0,signal=off -device \
virtconsole,chardev=charconsole0,id=console0 -enable-kvm -qmp unix:hypervisor.sock,server,nowait \
-device pci-bridge,bus=pcie.0,id=root1,addr=2,chassis_nr=1,shpc=on

Hot plugging N virtio disks

There is a limit of 30 devices per PCI bridge

for i in $(seq 1 30); do
    echo '{ "execute": "qmp_capabilities" }
    { "execute": "blockdev-add", "arguments":{"driver":"raw", "node-name": "disk'$i'", "file":      {"driver":"file","filename":"/dev/mapper/devmap" } } }
    { "execute": "device_add", "arguments": { "driver": "virtio-blk", "drive": "disk'$i'", "id": "virtio'$i'",  "bus": "root1", "bootindex": "'$i'", "addr": "'$(printf "0x%x" $i)'" } }' |
    socat - unix-connect:hypervisor.sock;
done

to detect the block device the pci bus MUST be re-scanned inside the VM

echo 1 > /sys/bus/pci/rescan

Reference

pcie_pci_bridge.txt

pcie.txt

Q35.pdf

@mcastelino
Copy link

Adding qemu docu links
https://github.com/qemu/qemu/blob/master/docs/pcie_pci_bridge.txt
https://github.com/qemu/qemu/blob/master/docs/pcie.txt

@mcastelino
Copy link

A slightly more user friendly command line which gives you a command prompt

sudo ./x86_64-softmmu/qemu-system-x86_64 -machine q35,accel=kvm,kernel_irqchip,nvdimm \
-m 256,maxmem=512M,slots=2 -smp 2 -nodefaults -rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-kernel ./vmlinuz.container  \
-append "reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1 init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false pci=lastbus=0 nivablecore=20G debug" \
-device virtio-serial-pci,id=virtio-serial0 -chardev stdio,id=charconsole0 \
-device virtconsole,chardev=charconsole0,id=console0 -nographic \
-object memory-backend-file,id=mem0,share,mem-path=./clear-containers.img,size=235929600 \
-device nvdimm,memdev=mem0,id=nv0 -no-reboot \
-net nic,model=virtio \
-net user \
-qmp unix:hypervisor.sock,server,nowait \
-device ioh3420,bus=pcie.0,id=rp3 -device pcie-pci-bridge,id=br3,bus=rp3 \

@gorozco1
Copy link

Adding q35 presentation link:
https://wiki.qemu.org/images/4/4e/Q35.pdf

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment