Debugging the kernel

Crashdump
DDB
Miscellaneous
References

Crashdump

when a kernel crashes, a reboot is inevitable but to preserve the current state of memory for later debugging, we should enable kernel crashdump. When enabled, before rebooting after occurring a crash on kernel-space, the operating system dumps it's current state into the swap device introduced earlier to the system. then after reboot and before entering the multi-user mode and before mounting the swap partition, the kernel look inside the swap partition for a possible crashdump. If there is any, kernel copies this dump into a specific directory defined by user.

Dump device

There are a few ways to define the swap partition to the system. Note that for a full memory dumps, you need a swap partition bigger than the current system's memory. When system is booting into multi-user mode, the values of dumpdev and dumpdir in /etc/rc.conf configuration file will be read by the system.

dumpdev: will define the swap device. The value of this configuration will be passed to dumpon command when system enters the single-user mode after a crash. possible values are:
- "AUTO" means system will choose a proper swap device from fstab records.
- "NONE" means no dump device
- "/dev/ada*" means the specific device as a dup devices
dumpdir: defines the directory which the system will copy the crash dump from the swap device to. Usually somewhere writable on disk like /var/crash but note this path should exists before a panic occurs.

Also a user can defines the dump device at runtime using dumpon command on multi-user mode. For example:

# dumpon -v /dev/ada0s3b

Since the dumpon command produces human readable error messages, We recommend to always check this manual way to ensure there is no error. For example the dump device should be a compatible swap typed partition, otherwise dumpon will prints error messages. to modify a current existing partition type to freebsd-swap parition, you can use gpart modify command. For example, to modify /dev/ada0s3 device run:

# gpart modify -i 2 -t freebsd-swap /dev/ada0s3
# gpart show

Producing Crashdump

To automatically create a crashdump after a panic, you can set the kern.coredump:

# sysctl kern.coredump=1

To manually create a crashdump from user-space you can either set debug.kdb.panic:

# sysctl debug.kdb.panic=1

Or enter ddb command prompt and request a dump command:

# sysctl debug.kdb.enter=1
(db) dump
(db) reboot

for more information, please read the DDB commands section.

Retrieving Crashdump

Now, when a panic occurs, (either on-demand or otherwise) system will dump it's memory into the dump device and reboots the system. On boot-up, it will copies the crash dump into the defined dumpdir by user. this can happen automatically or manually by user going into the single-user mode (or a live *NIX compatible system):

# savecore -vC /mnt/crash /dev/ada0s3b
# savecore -f /mnt/crash /dev/ada0s3b

The first command confirms there is a crashdump inside the given dump device and the second one actually stores it on the disk.

Example

The following commands shows how to remove all swap devices, modify a swap device, set the dump device and bring back the swap device on again and set to enter ddb prompt after a kernel panic occured:

# swapoff -a
# gpart modify -i 2 -t freebsd-swap /dev/ada0s3
# dumpon -v /dev/ada0s3b
# swapon /dev/ada0s3b
# sysctl kern.coredump=1
# sysctl debug.kdb.break_to_debugger=1
# sysctl debug.minidump=1

DDB

Intoduction

$ man 4 ddb

The ddb kernel debugger is an interactive debugger with a syntax inspired by gdb(1).

NOTE: Before reading further, please check kernel boot flags (/bootP/boot.config) no to contain flags like -m which mutes the kernel. for a list of boot flags please read man 8 boot.

How to enable DDB

To enable DDB, kernel should be compiled with the following options (although options DDB is only necessary, the rest is for debugging purpose):

makeoptions     DEBUG=-g  # Enable kernel debugging sybmbols
options KDB
options DDB               # built-in DDB support
options GDB               # enter GDB from DDB
options KDB_TRACE
options BREAK_TO_DEBUGGER # If need to enter DDB from console

How to enter DDB

There are a few ways to enter the DDB prompt:

with -d kernel option at boot time.
with the following sysctl option at runtime:
entering DDB on-demand:

sysctl debug.kdb.enter=1

To enter DDB using requesting kernel panic:

sysctl debug.debugger_on_panic=1
sysctl debug.trace_on_panic=1
sysctl debug.kdb.panic=1

On-demand entering DDB by pressing Ctrl+Alt+ESC hotkey.
Console break signal using <CR>~Ctrl+b
Also by defining panic shortcut key in the deafult keyboard layout. for more information please read How to panic on-demand when system is freezed.

How to enable remote debugging

The Host machine should have a copy of all the source files and the binary kernel with debugging symbols running on remote end. Also the remote machine should have special flags (0xC0) on UART module. On the remote machine edit /boot/device.hints (for more information read man 4 uart):

hint.uart.0.flags="0xC0"

Now on the Host machine, run kgdb with the binary of kernel:

# kgdb kernel
(kgdb) 
(kgdb) target remote /dev/cuau0
(db)

How to debug Dead-Locks

compile kernel using the following config (specially if you suspect the dead-lock is causing by VFS layer)

makeoptions 	DEBUG=-g
options 	INVARIANTS
options 	INVARIANT_SUPPORT
options 	WITNESS
options 	WITNESS_SKIPSPIN
options 	DEBUG_LOCKS
options 	DEBUG_VFS_LOCKS
options 	DIAGNOSTIC

When Dead-Locks occured, run the following commands:

ps
show pcpu
show allpcpu
show locks
show alllocks
show lockedvnods

How to panic on-demand when system is freezed

We can enable system to generate kernel panic by key while it's in an unresponsive state. using the following sysctl configuration. But beware the following method might only works when you're in a virtual tty console.

# sysctl machdep.enable_panic_key=1

To enable panic key, first list the available keyboard layouts in a virtual-terminal:

# kbdmap -p

To print the deafult keyboard layout:

# kbdmap -s

To select a new keyboard layout:

# kbdmap -d
keymap="us.iso.kbd"

To add a panic key to your default keyboard layout, edit /usr/share/syscons/keymaps/us.iso.kbd, and change the action for the esc key while pressing Alt+Ctrl+Shift to panic and to reboot while pressing Shift:

#                                                         alt
# scan                       cntrl          alt    alt   cntrl lock
# code  base   shift  cntrl  shift  alt    shift  cntrl  shift state
# ------------------------------------------------------------------
  000   nop    nop    nop    nop    nop    nop    nop    nop     O
  001   esc    boot  esc    esc    esc    esc    debug  panic   O

And then (optionally), reload the keyboard layout file using:

kbdcontrol -l /usr/share/syscons/keymaps/us.iso.kbd

Now, pressing the configure combination on keys above should produce a kernel panic.

DDB Commands

call <function>([<arg0>[,...]]): call any kernel function

cal boot # reboot the system
call cpu_reset() # reboot the system after calling sync() syscall

dump: creates a crashdump
reboot: reboots the system (usually after creating the crashdump)
trace [pid]: shows the current stack (if pid is not present) trace
where: shows the current stack
ps: display the current processes
continue: continue running kernel as normal
break <function> <address>: to set a break-point
del: delet all break-points
del <address-expression>: delete a specific breakpoint
show: displays various information
- show b: show already defined break-points
- show pcpu: shows the current process stack
- show allcpu: shows the current process stack for each CPU
- show all ifnets: shows the current kernel interfaces
- show msgbuf: shows the buffer of system message buffer
- show threads: kust like the ps command only for threads
s: single step into debugging
n: next command
x/<size><format> <address>: examine memory
w: word size
h: half-word size
b: byte size
x: hexadecimal format
d: decimal format
c: character format
s: buffer string
w/<size> <address> <byte-value> [<byte-value> [...]]: write memory
show reg: show register values
p $<register>: print a single register value
p $<register> <new-value>: assign new value to specific register
gdb: enter gdb debugging mode (requires kernel option GDB)
help

Miscellaneous

To check current kernel configuration:

# config -x /boot/kernel/kernel | egrep -i 'db|deb'

get to the kernel debugger via the serial console using the sequence spelled out for ALT_BREAK_TO_DEBUGGER (i.e., ~ ctl-b) (The FreeBSD Forums).
To make sysctl configuration permanent, edit /etc/sysctl.conf with the following syntax:
```
<key>=<value>
```

Martinfx/README.md