How to debug finagle tracing

tracing-tracer.md

Questions come up some times about why things get bad data. It is hard to pin things down, as maybe you can't modify the code, or even know where to look. Here's an example of how to track down what's making bad trace ids in finagle.

Firstly, download and extract byteman so you can use it.

$ (cd /tmp; wget -q http://downloads.jboss.org/byteman/3.0.5/byteman-download-3.0.5-bin.zip; unzip byteman-download-3.0.5-bin.zip)
$ export BYTEMAN_HOME=/tmp/byteman-download-3.0.5/

Now, start your finagle process byteman installed

$ java -javaagent:$BYTEMAN_HOME/lib/byteman.jar=boot:$BYTEMAN_HOME/lib/byteman.jar,listener:true -jar YOUR_JAR_HERE

Now, define and install a rule, such as dumping 5 lines of a call stack that invokes the trace id constructor

$ cat << 'EOF' > tracey.btm

RULE traceid
CLASS com.twitter.finagle.tracing.TraceId
METHOD <init>
AT ENTRY
IF true
DO traceStack("TraceId(" + $1 + ", " + $2 + ", " + $3 + ", " + $4 + ", " + $5 + ")", 5);
ENDRULE

EOF
$BYTEMAN_HOME/bin/bmsubmit.sh tracey.btm

Finally, watch the console where your new printlns are there!

Ex. In this case, I can see the code that was responsible for making a new trace id..

TraceId(None, None, 90cfdcf0036e87cd, None, Flags(0))com.twitter.finagle.tracing.TraceId.<init>(Id.scala:128)
com.twitter.finagle.tracing.Trace$.nextId(Trace.scala:150)
com.twitter.zipkin.query.FilteredHttpEntrypointTraceInitializer$.com$twitter$zipkin$query$FilteredHttpEntrypointTraceInitializer$$newRootSpan(FilteredHttpEntrypointTraceInitializer.scala:34)
com.twitter.zipkin.query.FilteredHttpEntrypointTraceInitializer$$anonfun$1.apply(FilteredHttpEntrypointTraceInitializer.scala:24)
com.twitter.zipkin.query.FilteredHttpEntrypointTraceInitializer$$anonfun$1.apply(FilteredHttpEntrypointTraceInitializer.scala:21)
  . . .