using rr with rust

rr-with-rust.md

using rust with rr

rr is a great debugging tool. it records a trace of a program's execution, as well as the results of any syscalls it executes, so that you can "rewind" while you debug, and get deterministic forward and reverse instrumented playback. it works with rust, but by default if you try it out, it could be pretty ugly when you inspect variables. if this bothers you, configure gdb to use a rust pretty-printer

rr is probably in your system's package manager.

usage

1. setup

• On some linux systems, you may need to echo 1 | sudo tee /proc/sys/kernel/perf_event_paranoid and sudo cpupower frequency-set -g performance to get better results.

2. compile with debug symbols

• this is on by default for tests.
• You can enable it for release builds in Cargo.toml if you need to:

  [profile.release]
  debug = 2
  

3. record a trace for deterministic replay / rewind

• rr record /path/to/executable, or
• rr record cargo test problem_test_7, or
• rr record -n cargo run (may be necessary when you can't control perf event paranoia)

replay

rr replay to open the debugger (rr is a bunch of functionality on top of gdb)

gdb cheatsheet

this gdb cheatsheet is helpful

tui mode

if you are used to using a debugger from an IDE, you may be much more comfortable while using GDB with the TUI. you can enter it by hitting Ctrl+x then Ctrl+a. for more info about using and configuring the TUI, check out its official docs. it can be configured to show or hide lots of interesting info.

navigation

feature	control	feature	control
step (goes into called functions)	s	reverse step	rs
next (doesn't go into called functions)	n	reverse next	rn
finish (current function)	f	reverse finish	reverse-finish
continue (to next breakpoint)	c	reverse continue	rc

inspecting state

info locals prints local variables (i lo for short, gdb is big on shortcuts) p <var name> prints contents

(rr) info locals
ret = Some = {Vec<u8>(len: 0, cap: 5120)}
start = 169602351
self = 0x7f8d07dfd9d0
key = &[u8](len: 1) = {165} 

list (l for short) can be used to view the contents of the current file, or another by providing the module name (tab-complete is your friend!)

(rr) list sled::tree::Tree::path_for_key
...
479                     debug_assert_ne!(not_found_loops, 10, "cannot find pid {} in path_for_key", cursor);
...

adding breakpoints

the line number provided matches the last file listed

(rr) break 479
Breakpoint 1 at 0x555929578ec4: /home/t/src/sled/src/tree/mod.rs:479. (2 locations)

this will be hit next time the code corresponding to this line is reached:

(rr) c
Continuing.

Thread 4 hit Breakpoint 1, sled::tree::Tree::path_for_key (self=0x7f8d07dfd9d0, key=&[u8](len: 1) = {...}) at src/tree/mod.rs:479
479                     debug_assert_ne!(not_found_loops, 10, "cannot find pid {} in path_for_key", cursor);

conditional breakpoints

(rr) list sled::tree::Tree::path_for_key                                                                                                                                                                                                                       
...
464         fn path_for_key(&self, key: &[u8]) -> Vec<(Node, CasKey<Frag>)> {
...
473             let mut not_found_loops = 0;
474             loop {
475                 let get_cursor = self.pages.get(cursor);
476                 if get_cursor.is_none() {
477                     // restart search from the tree's root
478                     not_found_loops += 1;
(rr) 
479                     debug_assert_ne!(not_found_loops, 10, "cannot find pid {} in path_for_key", cursor);
...
(rr) b 476 if not_found_loops > 5
Breakpoint 70 at 0x555929578e11: file src/tree/mod.rs, line 476.
(rr) i br
Num     Type           Disp Enb Address            What
70      breakpoint     keep y   0x0000555929578e11 in sled::tree::Tree::path_for_key at src/tree/mod.rs:476
        stop only if not_found_loops > 5 (host evals)
(rr) c
Continuing.
[New Thread 3140.3145]
[Switching to Thread 3140.3141]

Thread 3 hit Breakpoint 70, sled::tree::Tree::path_for_key (self=0x7f8d07dfd9d0, key=&[u8](len: 1) = {...}) at src/tree/mod.rs:476
476                 if get_cursor.is_none() {
(rr) i lo
get_cursor = core::option::Option::None
not_found_loops = 6
unsplit_parent = core::option::Option::None
path = Vec<(sled::tree::node::Node, sled::page::CasKey<sled::tree::frag::Frag>)>(len: 0, cap: 0)
cursor = 31
key_bound = Inc = {Vec<u8>(len: 1, cap: 1) = {165}}
self = 0x7f8d07dfd9d0
key = &[u8](len: 1) = {165}

adding watchpoints

watchpoints are able to break when a memory location is accessed, or given expression returns true. this is super powerfulfor speeding up the debugging process.

break whenever a variable of interest is accessed:

(rr) awatch self.root
Hardware access (read/write) watchpoint 4: self.root
(rr) c
Continuing.

Thread 2 hit Hardware access (read/write) watchpoint 4: self.root

Value = AtomicUsize = {v = UnsafeCell<usize> = {value = 0}}

Thread 2 hit Hardware access (read/write) watchpoint 4: self.root

Value = AtomicUsize = {v = UnsafeCell<usize> = {value = 0}} 
0x0000555929578d42 in sled::tree::Tree::path_for_key (self=0x7f8d07dfd9d0, key=&[u8](len: 1) = {...}) at src/tree/mod.rs:466
466             let mut cursor = self.root.load(SeqCst);

Lots more info available in the official docs:

• breakpoints
• rust caveats

Hi, thanks for guiding me in how to get rr working. I love it!

I couldn't get rr record cargo test to work, I had to first build a test binary and then run rr on it separately. You also want to configure rr to pretty-print rust code (rr replay -d rust-gdb).

I wrote a little wrapper that handles this sort of thing for you so you just call cargo rr test my_test and then cargo rr replay. https://crates.io/crates/cargo-rr