KGDB on Raspberry Pi 5 running Ubuntu using the Raspberry Pi Debug Probe

debug.md

TODO: try with linux-image-*-dbgsym package as well, maybe we can cut on the compile time.

Step 1: Compile

• Taken largely from this, though you can use your own kernel source. You can also do a cross-compilation, refer to the previous link for details on that
• specify the KERNEL (kernel_2712 in my case). Do make menuconfig instead of the raspi preset defconfigs
• Enable the KGDB options under Kernel Hacking. I don't remember exactly what flags you need, but the following in .config should set you up for something atleast
• • CONFIG_DEBUG_INFO=y
• • CONFIG_GDB_SCRIPTS=y
• • # CONFIG_DEBUG_INFO_REDUCED is not set
• CONFIG_KALLSYMS can also be set
• set a CONFIG_LOCALVERSION to give it a unique version, helpful if you have many kernels and kernel modules. This won't overwrite the existing ones.
• make -j6 Image.gz modules dtbs (I assume that we build natively on arm64. If you are cross compiling or building for 32 bit, please consult the guide linked above)
• sudo make -j6 modules_install - This will install the new kernel modules. Now onto some shenanigans.
• Check the version your newly built and installed kernel has by looking for a new directory under lib/modules which has your LOCALVERSION suffixed. This version string is important for creating the initramfs
• Copy the System.map and .config from your kernel source and copy it to /boot with the kernel version string suffixed, like /boot/System.map-$VERSION and /boot/config-$VERSION
• Make boot/dtbs/$VERSION/ and copy your Raspberry Pi's dtb in it (bcm2712-rpi-5-b.dtb for a Raspberry Pi 5, which will be in arch/arm64/boot/dts/broadcom/ in you kernel source.
• Make a symlink /boot/dtb-$VERSION which points to boot/dtbs/$VERSION/./<your-dtb>.dtb
• Make a symlink /boot/dtb which points to boot/dtbs/$VERSION/./<your-dtb>.dtb (most likely this already exists, so just update it)
• Take backups of /boot/vmlinuz* (any current or older vmlinuz, don't update the vmlinux symlink yet, make vmlinuz.old* symlinks to point to older kernels)
• Copy vmlinuz from your kernel source to /boot as /boot/vmlinuz-$VERSION, update the /boot/vmlinuz symlink to point to /boot/vmlinuz-$VERSION
• Copy <your-dtb>.dtb from arch/arm64/boot/dts/broadcom/ in you kernel source to /etc/flash-kernel/dtbs
• Create initramfs by doing sudo update-initramfs -c -k $VERSION (IMPORTANT: use MODULES=dep in /etc/initramfs-tools/initramfs.conf or in /etc/initramfs-tools/conf.d/ if it overrides the original conf. This will generate a smaller initramfs and will fit in /boot/firmware)
• Update /boot/initrd.img* symlinks accordingly (point /boot/initrd.img to /boot/initrd.img-$VERSION and use /boot/initrd.img.old for older versions)
• Check if /boot/firmware/vmlinuz is the same as /boot/vmlinuz-$VERSION. If not, take backups of existing /boot/firmware/vmlinuz* and copy the new vmlinuz into /boot/firmware
• Check if /boot/firmware/initrd.img is the same as /boot/initrd.img-$VERSION. If not, take backups of existing /boot/firmware/initrd.img* and copy the new initrd.img into /boot/firmware
• Add enable_uart=1 and enable_jtag_gpio=1 to /boot/firmware/config.txt under [all]
• Add nokaslr rodata=off maxcpus=1 to /boot/firmware/cmdline.txt
• Reboot and check uname -r to see if we have the new kernel running.

Step 2: The debug probe

• I suggest making a new directory.
• Copy the compiled kernel source from the Raspberry Pi to this directory on your main machine (main machine is the computer you run kgdb on, Raspberry PI is your debug target)
• Install gdb-multiarch and openocd
• Take your Debug Probe and connect its D port to the UART port on the Raspberry Pi 5. Connect the other end via USB to your main machine
• I largely followed this blog to get the OpenOCD interface and target scripts. I recommend you do teh same thing, except running GDB for now.
• After this, your Debug Probe might be powered up. Reboot the Raspberry Pi
• on the main machine, run gdb-multiarch <linux-source-copied-from-pi>/vmlinux
• on the openocd console, you will see 4 ports opened for the 4 CPUs. As we had set maxcpus=1 as a boot arg, we will take the port corresponding to CPU0, which is most likely :3333 (you can assume that the other CPUs are not being used. This is done to make debugging easier)
• In gdb, enter tar remote :<CPU port>. For me it was tar remote :3333
• type c (continue)
• Now it you raise SIGINT in GDB (by Ctrl-C), you will observer that the Raspberry Pi has completely halted (the cursor stops blinking if you have a monitor attached to it, or some other sign of a system halt) and on gdb you can see which function the kernel was in, and see the backtrace as well (along with symbols!!)
• Yay!!
• But if you try to debug a kernel module, you will see it can't find the symbols. No worries, we can fix this.
• NOTE: lx-symbols might fix this, but I haven't tried it yet.
• For example, if you are trying to debug the brcmfmac wifi driver (kernel module), what you need to do is look at /sys/module/<kernel-module-name>/sections/.text
• Then locate the associated *.ko file in your copied kernel source, and in gdb do add-symbol-file <path-to-kernel-module>.ko <address in .text>
• This will load the relevant symbols, and you can then debug the kernel module normally.

Step 3: lx-symbols

• Make sure you had CONFIG_GDB_SCRIPTS=y
• You should have a vmlinux-gdb.py in you kernel source root which is a symlink to scripts/gdb/vmlinux-gdb.py (if you don'e have it, create the symlink)
• You should add the kernel source root as a safe path by adding add-auto-load-safe-path <kernel-source>/scripts/gdb/vmlinux-gdb.py to ~/.config/gdb/gdbinit
• After this, in my case, gdb complained that linux.constants can't be imported (scripts/gdb/linux/constants.py.in was not generated)
• run make scripts_gdb from your kernel source root, this should generate the constants file.
• Reload GDB, connect to remote and then use lx-symbols
• It will load all the kernel modules at the right addresses