OS X v10.11 Kernel Debug Kit Read Me

Please Note: After installation, the Kernel Debug Kit will be available at: 

/Library/Developer/KDKs/

———————————————————————————————

The kernel file location has changed.

The kernel file location has moved to /System/Library/Kernels/kernel

DEVELOPMENT and DEBUG kernels

The OS X v10.11 Kernel Debug Kit includes the DEVELOPMENT and DEBUG kernel builds. These both have additional assertions and error checking compared to the RELEASE kernel. The DEVELOPMENT kernel can be used for every-day use and has minimal performance overhead, while the DEBUG kernel has much more error checking.

To allow development using the Kernel Development Kit on OS X v10.11, you will need to modify the security configuration with the following steps:

1. Reboot to the Recovery System by restarting your Mac and hold down the Command and R keys at startup.
2. From the Utilities menu, select “Terminal”.
3. Type, “csrutil disable” to disable System Integrity Protection, “csrutil enable” to reenable System Integrity Protection.
4. Restart your Mac.

To install these kernels, copy the kernel.development or kernel.debug file from the KDK to /System/Library/Kernels, and add “kcsuffix=development” (or “kcsuffix=debug” as appropriate) to your boot args.  Please note, you will need to rebuild your kextcache to use the new kernel for debugging. This can be done by running the “kextcache -invalidate” command against the target volume as follows:

     sudo kextcache -invalidate /Volumes/<Target Volume>
     sudo reboot

To return to the Release kernel, delete the kernel.development/kernel.debug file, remove the “kcsuffix” boot-args and run the following:

     sudo rm /System/Library/Caches/com.apple.kext.caches/Startup/kernelcache.de*
     sudo rm /System/Library/PrelinkedKernels/prelinkedkernel.de*
     sudo kextcache -invalidate /
     sudo reboot

Two Machine Debugging In OS X v10.11

Finding Symbols Automatically

lldb will search /Library/Developer/KDKs/ as well as any local directories indexed by Spotlight.

You invoke lldb by the simple command:

     lldb

Alternatively, you can specify the kernel file on the command line.

     lldb /Library/Developer/KDKs/<KDK Version>/System/Library/Kernels/kernel

To attach, you either use kdp-remote for a live connection, or file -c for a coredump.

     kdp-remote {name_or_ip_address}
     file -c {path_to_coredump}

Two machine debugging is not supported via USB Ethernet.
Two machine debugging is not supported via wireless.

Network Debugging

The default setting for two machine debugging is as follows:

     sudo nvram boot-args="debug=0x146"
     sudo reboot

This assumes you are using the first Ethernet port, en0.

If you are running the development kernel, this would be enabled as follows:

     sudo nvram boot-args="<existing boot-args> kcsuffix=development"
     sudo kextcache -invalidate /
     sudo reboot

If you are on a machine with multiple Ethernet ports (e.g. a MacPro), use ifconfig to determine the correct port. For instance, add "kdp_match_name=en1" if using port en1.

If you are on a machine without built-in Ethernet, you can use one of the following: 

Thunderbolt to Ethernet adaptor
Ethernet port on a Apple Thunderbolt display
Use the ifconfig command to determine the appropriate port. The output of ifconfig will show all ports. You want to specify a port with a valid inet option. For example if this was the ifconfig output:

     en1: flags=8963 mtu 1500
         options=60
         ether 32:00:13:ee:19:e0
         media: autoselect 
         status: inactive
     en2: flags=8863 mtu 1500
         options=10b
         ether 40:6c:8f:5b:a2:96
         inet6 fe80::426c:8fff:fe5b:a296%en2 prefixlen 64 scopeid 0x4
         inet6 2620::1b07:114:426c:8fff:fe5b:a296 prefixlen 64 autoconf
         inet6 2620::1b07:114:88d6:bbba:7ac9:b0a7 prefixlen 64 autoconf temporary
         inet 10.128.19.135 netmask 0xfffff800 broadcast 10.128.23.255
         nd6 options=1
         media: autoselect (1000baseT )
         status: active

Port en2 is the valid port in this example. You would specify this as follows:

     sudo nvram boot-args="debug=0x146 kdp_match_name=en2"
     sudo reboot

Remember, neither USB nor Wireless work for two machine debugging.

Firewire Debugging

The default setting for two machine debugging is as follows:

     sudo nvram boot-args="debug=0x146 kdp_match_name=firewire fwkdp=0x8000"
     sudo reboot

Non-built-in FireWire controllers are specifically excluded from working for both FireWireKDP and FireWireKPrintf. This is intentional. However there is an easy way to enable them.

To enable non-built-in interfaces for FireWireKDP, add the "fwkdp=0x8000" boot-arg. For FireWireKPrintf, use "fwkpf=0x8000".

However, this support comes with a caveat. Once the FireWire controller has been "used" for either FireWireKDP or FireWireKPrintf, it must not be unplugged or the machine will likely panic. So, if you're using a Thunderbolt Display or a Thunderbolt to FireWire Adapter, you should just leave them connected until you disable the "0x8000" boot-arg and restart.

If you are on a machine without built-in firewire, you can use one of the following: 

Thunderbolt -> Firewire adaptor
Firewire port on a Apple Thunderbolt display
On the host machine, open a terminal window and type the following:

     fwkdp

Leave that window open. Now in a second window, you can invoke lldb and issue the following command:

     kdp-remote localhost

The fwkdp redirector software redirects kdp packets as if the host machine was the panicked machine. Saving a coredump is done locally on the host, e.g. in lldb:

     (lldb) sendcore 1.2.3.4 # IP address does not matter for firewire debugging
     (lldb) detach

Reporting Kernel Panics or System Hangs on OS X v10.11

Simple Technique

You may restore your system to default settings by deleting your nvram boot-args variable:

     sudo nvram -d boot-args
     sudo reboot

After doing this, your system will reboot after a panic, after which you may file a bug report at http://bugreport.apple.com. Include a full system profiler report (not basic) and any panic logs from the /Library/Logs/DiagnosticReports directory. 

Note that the automatic prompt to report a panic after you reboot is not sufficient. That goes into a giant database which aggregates panic reports and produces a bug report eventually. This automatically generated bug report may not have enough information for us to diagnose the issue. By following the instructions above and always filing a bug report for your panics, you make it possible for us to better diagnose your issue

Coredumps

If you have another Mac, you can try to set up the second Mac as a coredump server.

For a coredump, you need an Ethernet connection (airport will not work, nor will USB Ethernet) from the panicking computer to another machine, or to a router. (Any Mac will work as a coredump server; you just need a gigabyte or so of free space per coredump.) On the server (non-panicking) machine run the following commands:

     sudo mkdir /PanicDumps
     sudo chown root:wheel /PanicDumps
     sudo chmod 1777 /PanicDumps
     sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.kdumpd.plist

On the client (panicking machine):

     sudo nvram boot-args="debug=0xd44 _panicd_ip=10.0.40.2"
     sudo reboot

where "10.0.40.2" is replaced by the IP address of the server. (You must reboot the client for the nvram settings to take effect.)

Note: if you are using a machine without built-in networking, or if you have a MacPro, or if you want to use firewire, see Two Machine Debugging for details.

If you hang, NMI the machine by hitting the buttons Left-⌘ + Right-⌘ + Power. This will generate a coredump file on the server machine in the directory /PanicDumps.

If you panic, the coredump file will be generated automatically on the server machine in the /PanicDumps directory.

Compress the coredump file saved to /PanicDumps, and attach that to the bug report you file. Coredumps are mostly zeros, so they compress very well (an average coredump of 500Mb will compress down to 30Mb or so, so compress it before trying to attach it to a bug report.) Please file a new bug report for each panic.

To remove this behavior from the client machine, run the following:

     sudo nvram -d boot-args
     sudo reboot

You must reboot to make new nvram settings take effect.

Please note, coredumps may contain private data.

Hangs

The best information is acquired by the sysdiagnose command. The sysdiagnose tool gathers system diagnostic information helpful in investigating system performance issues. A great deal of information is harvested, spanning system state and configuration. sysdiagnose needs to be run as root. sysdiagnose can be triggered upon pressing a special key chord; this is currently Control-Option-Command-Shift-Period. There is a man page with additional information available; open a terminal window and type the following:

     man sysdiagnose