Debugging MongoDB with gdb

gdb is a command line tool that you can use to examine the state of a (1) terminated or (2) running C or C++ process.

Background: Debugging a terminated process

A process can terminate (uncleanly) for a few reasons, such as triggering a segmentation fault, failing an invariant, or throwing an uncaught exception.

A binary can be compiled so that when a process terminates (uncleanly), the process produces a core dump, which is a file containing a "frozen" record of the state of all threads that existed in the process at the time it terminated.

You can use gdb as "viewer" of this core dump file to examine the frozen state. For example, you can use gdb on the core dump file to:

list all threads
in each thread, show the current backtrace
for each frame in each thread's backtrace, print the values of the local variables in that frame
print values of the global variables

Though having a core dump is useful to examine the "end state" of a process that terminated uncleanly, it doesn't tell you how the process reached the "end state."

Examining a core dump with gdb is most useful for understanding why a process terminated unexpectedly in the wild.

Background: Debugging a running process

If you want to trace how a process reaches a particular state, it’s more useful to execute the process through gdb.

This allows you to use breakpoints to specify at what points the process should "freeze."

Once the process has hit a breakpoint and frozen, you can use gdb to examine its current state, just as you would examine a core dump.

The difference is that, once you are done examining the frozen state, you can use gdb to "unfreeze" the process so that the process runs until the next breakpoint is hit.

Stepping through code

If you're interested in stepping through some region of code, it would be rather tedious to specify a breakpoint for each line. For this reason, gdb provides "syntactic sugar" commands that, once a breakpoint is hit, allow you to advance the process by one statement ("step") or one line ("next") at a time.

So, you only need to specify a breakpoint on the first line of the region.

Using gdb to step through a running process is most useful when you can reproduce some behavior locally, but want to trace the code paths and changes in the process’s state that lead to that behavior.

Setting up your system to debug MongoDB with gdb

To debug with gdb, you have to

install gdb on your local machine
compile MongoDB binaries for use with gdb (that is, pass flags to the compiler to produce binaries that contain debug symbols).

This tutorial assumes you are using a linux-based operating system (not OS X or Windows; the author uses Ubuntu 16.04), and assumes you know (or are comfortable looking up) how to:

do basic commands on your machine's terminal (edit files, change file permissions, execute a binary)
compile MongoDB binaries on your machine (directly with scons or through ninja)

Install gdb (and gdbserver)

If you are a MongoDB employee, you should already have installed the MongoDB toolchain, and you should be using either the gcc or clang compiler in the toolchain to compile your MongoDB binaries.

Using the compilers (gcc, clang) and debugger (gdb) from the toolchain ensures that the versions of your compiler and debugger are compatible with each other and support the pretty printers for MongoDB types (more on those below).

How to install the MongoDB toolchain

Follow the directions in this email to [email protected] to install the MongoDB toolchain for your operating system.

How to compile with the gcc or clang in the toolchain

If you are compiling directly with buildscripts/scons.py, then to compile with the gcc in the toolchain, pass --variables-files=etc/scons/mongodbtoolchain_gcc.vars to your scons invocation, e.g.
$ ./buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_gcc.vars mongod mongos mongo
Similarly, to compile with the clang in the toolchain, pass --variables-files=etc/scons/mongodbtoolchain_clang.vars:
$ ./buildscripts/scons.py --variables-files=etc/scons/mongodbtoolchain_clang.vars mongod mongos mongo
If you are compiling through ninja, you will have to rebuild your .ninja executable with the --variables-files argument. See the ninja github repo for directions on recompiling .ninja.

Once you have the toolchain set up, you should see a copy of gdb and gdbserver in your /opt/mongodbtoolchain/gdb/bin directory:

$ ls /opt/mongodbtoolchain/gdb/bin
gcore  gdb  gdbserver

You should now either

add /opt/mongodbtoolchain/gdb/bin to your $PATH (see here for a quick refresher on editing your $PATH), or
create an alias for "gdb" and "gdbserver" to point to the one in the toolchain (see here for a quick refresher on creating an alias).

If you are not a MongoDB employee, you can install gdb and gdbserver using your system's package manager. For example:

$ sudo apt-get install gdb gdbserver

The gdb and gdbserver executables should automatically be placed in your /usr/bin directory, which should already be in your $PATH.

You will have to ensure on your own that the version of gdb you install is compatible with the version of gcc or clang used to compile the binaries.

Compile MongoDB binaries with debug symbols

Once you have installed gdb, the next step is to compile your binaries with debug symbols. Fortunately, MongoDB has set up our build system (SCons) to automatically compile with debug symbols.

Quick background on debug symbols

When you compile source code, the compiler builds a symbol table that maps variable names, function names, and literals to their type, scope, and other information.

You can use gdb on any binary compiled from C or C++ source code - unit test, dbtest, mongod, mongos, mongo, etc.

How to use gdb

Examining a core dump

Pass gdb the core dump of a process that has terminated, plus the binary that was used to launch the process

Attach gdb to a hung (but still running) process

Similar to debugging a core dump

Execute a binary entirely through gdb

Some basic gdb commands to get you started

Examine frozen state

info threads

backtrace or bt

Step through code

run or r

step or s

next or n

continue or c

What's next?

Become a gdb power user

Install gdb pretty-printers for the C++ STL types, Boost types, and MongoDB types

Pretty printers produce user-friendly string descriptions of types.

These all involve installing Python libraries and registering them with gdb through your ~/.gdbinit.

C++ Standard Library Data structures (vectors, stacks, etc): https://sourceware.org/gdb/wiki/STLSupport
Boost objects: https://github.com/ruediger/Boost-Pretty-Printer
MongoDB: https://github.com/mongodb/mongo/blob/master/buildscripts/gdb/mongo.py

See all gdb commands and command-line options

http://www.yolinux.com/TUTORIALS/GDB-Commands.html

Step through a multithreaded process

Use gdbserver

Compile the binaries with --gdbserver so that they hang rather than terminate

This is mainly useful for replica sets and sharded clusters:

Compile the binaries with --gdbserver, then start the processes as normal. If a breakpoint is hit or the process would have terminated (due to a segmentation fault, invariant failure, or uncaught exception), instead of terminating it will launch a "gdbserver" process and hang. You can then use gdb to attach to the "gdbserver" that is attached to the hung process.

EshaMaharishi/debugging_mongodb_gdb.md