Skip to content

Instantly share code, notes, and snippets.

@mikesmullin

mikesmullin/vbox_vm_core_analysis.md

Last active May 7, 2025 19:49
Show Gist options
  • Save mikesmullin/8163354758e274e0b015b122177d0282 to your computer and use it in GitHub Desktop.
Save mikesmullin/8163354758e274e0b015b122177d0282 to your computer and use it in GitHub Desktop.
Download ZIP
HOWTO: VirtualBox VM Pause, Core-dump, and Root-Cause Analysis (w/ Volatility3)
Raw
vbox_vm_core_analysis.md

HOWTO: Snapshot & Analyze a VBox VM

VBoxManage controlvm "k8s-ctl3" pause

VBoxManage debugvm "k8s-ctl3" dumpvmcore --filename ~/k8s-ctl3-core.dump

user@FAFB09C2:~$ VBoxManage debugvm "k8s-ctl3" getregisters --cpu 0 rip
rip = 0xffffffff81ddde2b
user@FAFB09C2:~$ VBoxManage debugvm "k8s-ctl3" getregisters --cpu 1 rip
rip = 0xffffffff81ddde2b
user@FAFB09C2:~$ VBoxManage debugvm "k8s-ctl3" getregisters --cpu 2 rip
rip = 0xffffffff81ddde2b
user@FAFB09C2:~$ VBoxManage debugvm "k8s-ctl3" getregisters --cpu 3 rip
rip = 0xffffffff817e4626

# whoa cowboy! need to dump the profile first...
#python2 volatility/vol.py -f k8s-ctl3-core.dump --profile=LinuxUbuntu1604x64 pslist

# from: https://www.iblue.team/memory-forensics-1/volatility-plugins/build-custom-linux-profile-for-volatility
# vol3

apt install -y python3-pip
pip3 install volatility3
vol

vol -f k8s-ctl3-core.dump banners

Volatility 3 Framework 2.11.0
Progress:  100.00               PDB scanning finished
Offset  Banner

0x73a00200      Linux version 5.15.0-131-generic (buildd@lcy02-amd64-057) (gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #141-
Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 (Ubuntu 5.15.0-131.141-generic 5.15.168)

# on replica system, export profile

lsb_release -cs # jammy

nano /etc/apt/sources.list.d/ddebs.list

deb http://ddebs.ubuntu.com jammy main restricted universe multiverse
deb http://ddebs.ubuntu.com jammy-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com jammy-proposed main restricted universe multiverse

wget -O - http://ddebs.ubuntu.com/dbgsym-release-key.asc | sudo apt-key add -

apt update
apt install linux-image-5.15.0-131-generic-dbgsym

# should reboot!?
sudo reboot

git clone https://github.com/volatilityfoundation/dwarf2json.git
cd dwarf2json.git
apt install -y golang-go
go install

echo "export PATH=\$PATH:$HOME/go/bin" >> ~/.profile
source ~/.profile

# need 8gb ram for this, enable swap
sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

dwarf2json linux --elf /usr/lib/debug/boot/vmlinux-5.15.0-131-generic --system-map /boot/System.map-5.15.0-131-generic > Ubuntu22.04-5.15.0-131-generic.json
zip ubuntu.zip *.json
scp ubuntu.zip user@lappy:~/

# back to investigation

# get python path
$ python3 -c "import volatility3; print(volatility3.__path__)"
/home/user/.pyenv/versions/3.11.5/lib/python3.11/site-packages/volatility3
# move profile json into this path
mv Ubuntu22.04-5.15.0-131-generic.json ~/.pyenv/versions/3.11.5/lib/python3.11/site-packages/volatility3/symbols/linux/Ubuntu22.04-5.15.0-131-generic.json
# verify volatility3 sees it
vol isfinfo

# (finally) attempt to use
vol -f k8s-ctl3-core.dump linux.pslist
# (working!)

# valid cmds on linux core dump
vol -f k8s-ctl3-core.dump linux.check_afinfo
vol -f k8s-ctl3-core.dump linux.check_creds
vol -f k8s-ctl3-core.dump linux.check_idt
vol -f k8s-ctl3-core.dump linux.check_modules
vol -f k8s-ctl3-core.dump linux.check_syscall
vol -f k8s-ctl3-core.dump linux.ebpf
vol -f k8s-ctl3-core.dump linux.elfs # mem ranges for running procs + dlls. use FOURTH
vol -f k8s-ctl3-core.dump linux.envars
vol -f k8s-ctl3-core.dump linux.hidden_modules
vol -f k8s-ctl3-core.dump linux.iomem
vol -f k8s-ctl3-core.dump linux.keyboard_notif
vol -f k8s-ctl3-core.dump linux.kmsg # dmesg log. very useful. use THIRD! or FIRST! 
vol -f k8s-ctl3-core.dump linux.kthreads
vol -f k8s-ctl3-core.dump linux.library_list
vol -f k8s-ctl3-core.dump linux.lsmod
vol -f k8s-ctl3-core.dump linux.lsof
vol -f k8s-ctl3-core.dump linux.malfind
vol -f k8s-ctl3-core.dump linux.mountinfo
vol -f k8s-ctl3-core.dump linux.netfilter
vol -f k8s-ctl3-core.dump linux.pagecache
vol -f k8s-ctl3-core.dump linux.pagecache
vol -f k8s-ctl3-core.dump linux.pidhashtable
vol -f k8s-ctl3-core.dump linux.proc # dump mem ranges for all proc. use SECOND!
vol -f k8s-ctl3-core.dump linux.psaux # most useful way to get pslist. use FIRST!
vol -f k8s-ctl3-core.dump linux.pslist # okay
vol -f k8s-ctl3-core.dump linux.psscan # not so useful
vol -f k8s-ctl3-core.dump linux.pstree # crashes
vol -f k8s-ctl3-core.dump linux.ptrace # no output in this case
vol -f k8s-ctl3-core.dump linux.sockstat # like netstat! useful
vol -f k8s-ctl3-core.dump linux.tty_check

linux.kmsg found this: smoking gun

│kern emerg 60454.363030 CPU(1) watchdog: BUG: soft lockup - CPU#1 stuck for 11291s! [swapper/1:0]

Grok AI

from: https://x.com/i/grok?conversation=1920143288377708877

Root-Cause Analysis

What Caused the RCU Stall?

  • Virtio Network Driver (virtio_net): The CPU was stuck in virtqueue_enable_cb_delayed within the Virtio network driver (virtio_net). This function manages Virtio queues, which handle network packet transmission in VirtualBox. The stall suggests the Virtio driver was unable to complete its operation, likely due to a deadlock, infinite loop, or resource contention in the driver or VirtualBox’s Virtio emulation.
  • TCP Keepalive Timer: The issue originated from the tcp_keepalive_timer, which was sending TCP keepalive packets. Kubernetes control-plane nodes (like k8s-ctl3) rely heavily on TCP connections (e.g., for kube-apiserver, etcd, and pod communication). A large number of TCP connections or a misconfiguration might have overwhelmed the network stack.
  • High Network Load: The stack trace shows the packet being processed through the network stack (tcp_xmit_probe_skb, ip_queue_xmit, dev_queue_xmit), indicating high network activity. Given the VM’s role as a Kubernetes control-plane node, it’s likely handling a large number of API requests, pod communications, or etcd traffic, leading to network saturation.
  • VirtualBox Virtio Emulation: VirtualBox’s Virtio network emulation might have a bug or performance bottleneck, especially under high load. The virtqueue_enable_cb_delayed function getting stuck suggests VirtualBox failed to handle the Virtio queue properly, possibly due to contention between the guest and host.
  • RCU Stall Consequences: The RCU stall prevented the CPU from making progress, blocking RCU callbacks and softirqs (e.g., network-related softirqs). This caused the guest OS to become unresponsive, as critical kernel operations couldn’t complete. The high number of queued RCU callbacks (q=3931843) indicates a severe backlog, exacerbating the issue.

Volatility 2

I could not get this to work. python2 rot?

#zip Ubuntu_$(uname -r)_profile.zip ./module.dwarf /boot/System.map-$(uname -r)

$ zip volatility/plugins/overlays/linux/Ubuntu-5.15.0-131-generic.zip tools/linux/module.dwarf boot/System.map-5.15.0-131-generic
  adding: tools/linux/module.dwarf (deflated 91%)
  adding: boot/System.map-5.15.0-131-generic (deflated 80%)

# ...

$ ls -lh ~/volatility/plugins/overlays/linux/
Ubuntu-5.15.0-131-generic.zip
# verify contents
$ unzip -l ~/volatility/plugins/overlays/linux/Ubuntu-5.15.0-131-generic.zip
$ python2 ~/volatility/vol.py --plugins=/home/user/volatility/plugins/ --info | grep Linux

# re-attempt pslist
$ python2 ~/volatility/vol.py --plugins=/home/user/volatility/plugins/ -f k8s-ctl3-core.dump --profile=LinuxUbuntu-5_15_0-131-genericx64 pslist

# if it fails here, maybe profile/symbols are wrongly built!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment