If you need to debug a panic on a remote FreeBSD system, but don't want to transfer large coredumps, and can't afford the downtime caused by KDB interrupting the boot process, a text dump may be the answer. textdump(4) is the kernel dump facility, which allows you to see the crash information in a text format. It also allows for configuration of exactly what information is captured, by scripting input as if you were at the KDB prompt on the console.
Textdump has certain requirements. First, you must be running FreeBSD 7.1 or higher, and your kernel config must include the following options.
options DDB
options KDB
options KDB_UNATTENDED
options KDB_TRACE
Textdump also relies on the dumpdev and savecore features, both of which are enabled in /etc/rc.conf. Dumpdev can be set to "AUTO" to use the first configured swap device, or you can set the device name manually to your preferred swap partition. Do not specify a UFS filesystem device name or path here; only swap can be used, as the kernel cannot reliably access filesystems after a panic. This setting is also used by savecore at boot to extract the saved dump from swap.
dumpdev="AUTO"
ddb_enable="YES"
Also note that you must either ensure that the default target directory for savecore ( /var/crash ) exists, or over-ride the default setting dumpdir in /etc/rc.conf.
Next, take a look at /etc/ddb.conf, in particular this section. The script line provides the scripted commands that will be issued through KDB at the next panic:
# kdb.enter.panic panic(9) was called.
script kdb.enter.panic=textdump set; capture on; run lockinfo; show pcpu; bt; ps; alltrace; capture off; call doadump; reset
You can enter any series of valid ddb(4) commands, the output of which will be written to the text dump. The last entry, reset, will initiate the reboot.
Once you have a kernel compiled with DDB/KDB enabled, and the required settings in rc.conf and ddb.conf, reboot to load the new kernel and configuration. You can double-check that your settings are in place, and look for the following sysctl entries (kdb.enter.panic should be identical to the script setting in /etc/ddb.conf).
# sysctl -a|grep kdb
kdb.enter.panic=textdump set; capture on; run lockinfo; show pcpu; bt; ps; alltrace; capture off; call doadump; reset
machdep.kdb_on_nmi: 1
You'll need to tell the system that you want to do a text dump. Put this line into your /etc/sysctl.conf file:
debug.ddb.textdump.pending=1
The first time you do it, make sure it's set.
sysctl -a | grep textdump
Now, instead of a "ddb>" prompt after a panic, you will see the console quickly scroll through the output of your script commands, and then automatically reboot.
Once the machine boots, you should see that "savecore" runs and extracts the saved text dump.
savecore: writing core to textdump.tar.0
The text dump will be stored in /var/crash (or whichever directory you set dumpdev to in rc.conf). Multiple dumps will be stored in a numbered format. The dump output itself is stored as a tar file.
etbwmgr# ls -l /var/crash
-rw-r--r-- 1 root wheel 2 Feb 5 16:31 bounds
-rw------- 1 root wheel 443 Feb 5 16:31 info.0
-rw------- 1 root wheel 69632 Feb 5 16:31 textdump.tar.0
etbwmgr# tar -xvf /var/crash/textdump.tar.0
x ddb.txt
x config.txt
x msgbuf.txt
x panic.txt
x version.txt