GDB (GNU Debugger)

Quick Reference

gdb [file name]
b [breakpoint line #]
start
c (go to first breakpoint)
p [var name] (print a variable value)

Notes

be sure to compile correctly to allow debugging info. creation, via gcc, i.e.,

gcc -g hello.c

or with a special variant of make (which you can put in your makefile)

CFLAGS="Wall -g" make [source_filename]
type gdb [filename] to use gdb with a program
type start in the interactive gdb prompt to start stepping through the program
type list to show entire program
GDB has a built-in Python interpreter, allowing for scripting of debugging sessions

(gdb) python
import os
print (“my pid is %s” % os.getpid())

python print(gdb.breakpoints())  // list breakpoints
python print(gdb.breakpoints()[0].location)  // look at an individual breakpoint
python gdb.Breakpoint(‘7’)  // create a new breakpoint at line 7

GDB Commands

keys	Description
$ gdb o.out	run gdb on program
ctrl + x + a	enter/exit TUI (text user interface) mode
b [file:line # or function name]	create a breakpoint
run [args]	begin running the program w/optional args
bt	backtrace
list(-)	when the program is stopped, shows the next(/previous) 10 lines of code after the breakpoint
next	runs the program line by line, limited to the current scope
step	runs the program line by line, following execution through function calls throughout the program
cont	continue running the program
p or print [variable_name ]	examine the current state of a variable
set [variable] = [value]	set the state of a variable
watch [variable]	set a watchpoint, which notifies you when a value is written to the variable
kill	stop a paused program
quit	quit gdb
thread apply all bt	dump a backtrace for all threads
gdb --batch --ex r --ex bt --ex q --args	run program so that if it crashes, you get a backtrace
cd/pwd/make	function just as in the shell
shell	quickly start a shell to do other things
clear	clear a breakpoint
info break/info watch	show info about breakpoints and watchpoints
attach pid	attach to a running process so you can debug it
detach	detach from the process

GDB TUI Mode Commands

keys	description
ctrl + L	repaint screen (when text displays over text)
ctrl + x + 2 (numpad)	cycle through window (assembly & registers)
ctrl + x + 1	cycle back through window (source code)
tui reg float	switch from general-purpose to floating registers
ctrl + p / ctrl + n	previous/next command in history
help	help menu

Setting Watchpoints

you can only set a watchpoint for a variable when it is in scope
to watch something within another function or inner block, first set a breakpoint inside that scope and then, when the program pauses there, set the watchpoint

Core Dump Settings Check

ulimit -c shows max size of core dumps created cat /proc/sys/kernel/core_pattern https://www.kernel.org/doc/Documentation/sysctl/kernel.txt

Integrated Python

command	description
`python print (gdb.breakpoints())`	print all breakpoints
`python print (gdb.breakpoints()[0].location`	print an individual breakpoint

Reversible Debugging

intermittent bugs are hard to reproduce and troubleshoot
if a bug destroys the stack, core dumps are useless for debugging purposes

debugging intermittent bugs - set up a loop to keep running the program over and over until the bug shows up

enable reversible debugging
when program faults, step back a bit
get the context of the bug

you can set breakpoints and issue commands when those breakpoints are hit

ex. --

(gdb) b 7
(gdb) b main
(gdb) b exit.c:32
(gdb) command 3
run
end
(gdb) command 2
record
continue
end

(gdb) set pagination off
(gdb) run

program will run repeatedly until the bug is triggered, then gdb will pause the program, at which point you can take a step back and see what happened

(gdb) reverse-stepi

you now have a stack that works once again and can look as the disassembly of the code

disas

you can print variables in their current state at the point you stepped back to, such as the stack pointer, and watch a watchpoint/data breakpoint, looking backwards until the data in the memory location watched changes, which will show you what ruined the stack and caused the bug, ideally.

(gdb) print $sp
(gdb) print *(long**) 0x7ffffffffdc98
=> 3 = (long *) 0x5e4c5d00
(gdb) watch *(long**) 0x7ffffffffdc98
(gdb) reverse-continue

Simple Use-Case Walk-Through

First, I setup a breakpoint (3 in the example below) where the program exits successfully, 
i.e. without error which is uninteresting to us. To do this I do:

(gdb) command 3
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
run
end
I then created another breakpoint for when things go wrong. If this happens, I want 
recording to start (so I can do reversible debugging) and then continue:

(gdb) command 2
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
record
continue
end
When I run this, each time the program exits successfully it just tries again until it 
fails. When it does fail with the segfault error I am in GDB and able to look around 
so I can just reverse-step i and then I’m back into a sensible stack which I can 
inspect and understand.

I can then inspect the stack, set an appropriate watchpoint and use reverse-continue 
to find out at what point the location in memory is changed so I can find the 
offending part of the code. At this point I can explore the local variables and 
find out what happen.

To reiterate what I've done: by using a couple of features of GDB I took an 
impenetrable bug with no useful coredump and made it so I can dig into the 
moment the error occurs.

Resources

https://www.cprogramming.com/gdb.html

https://undo.io/resources/cppcon-2015-greg-law-give-me-15-minutes-ill-change/

http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html

vonnenaut/gdb.md