Adrian Cole codefromthecrypt

This is a reaction post to Misunderstanding "Open Tracing" for the Enterprise by @jkowall

This is opinions with citations (imagine that!). This is not wikipedia. sorry. I didn't run this by anyone, Jonah or otherwise. I do not represent OpenTracing or OpenCensus (or my employer or whatever you might think) in this view. I will give critical thoughts on both from a technical view as people complained to me mostly about lack of details. I do have "a dog in the race" but it isn't what you might think. Yes, I'm the primary maintainer of Zipkin, but my goal is not to disparage anything rather to keep the community healthy with options that exist and free from the suffering caused in my opinion by complete lack of technical view on what things do. Particularly, this is dangerous in interop, and I'll get to that.

I have experience with both tools. Though I left within months, I was implicated in the beginning of OpenTracing. I still mainta

The number one problem to solve in tracing is propagation, at least carrying trace identifiers from one side of your process to the other. Usually, some thread local state is involved with this. In I'd say most libraries apart from OpenTracing, this is a separate functionality than timing things. For example, you put variables in scope with one api and you do timing with another. It might look like this.

span = startSpan();
try (Scope scope = propagation.scope(span.ids())){
  // work here can see the span ids via something like Trace.ids() 
} finally {

I've been party to some pretty interesting conversations around Zipkin's ID format and UUIDs. Seems interesting to share, since some of this stuff is neat, if geeky.

So, we love UUIDs! Many people use UUID format to pass around things unlikely to ever clash. These are used for long-term retrieval and other handy things. A lot of correlation IDs are UUIDs, too.

What about distributed tracing? Trace IDs should be UUID, too! To this I answer.. maybe?

Firstly, there are a lot of systems inspired by the dapper paper which hints how

I have heard a number of APMs create "spans" (distributed tracing lingo for an operation) and aggregate them for reasons like latency metrics.

In a way, Zipkin does this. The ever popular service dependency diagram is an aggregated view of parent/child links between services with the number of calls between them added for color.

The biggest issue with using a tracing api to back metrics is that most of the time, tracing is sampled (like 1 out of 1000). Sampling is done to reduce costs or prevent a surge of traffic from taking out the

	while true; do curl -s http://192.168.99.100:9411/api/v2/spans -H'Content-Type: application/json' -H 'Expect:' -d'[{
	"traceId": "4d1e00c0db9010db86154a4ba6e91385",
	"parentId": "86154a4ba6e91385",
	"id": "4d1e00c0db9010db",
	"kind": "CLIENT",
	"name": "get",
	"timestamp": 1472470996199000,
	"duration": 207000,
	"localEndpoint": {
	"serviceName": "frontend",

	# this returns all of the annotation values, not just the http.url
	# for more.. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html
	curl -s 'localhost:9200/zipkin-*/span/_search' -d'{
	"_source": false,
	"aggs": {
	"binaryAnnotations": {
	"nested": {
	"path": "binaryAnnotations"
	},
	"aggs" : {

	# substitute 2017-01-18 for the correct date and foo and bar for the two services names you're interested in
	curl -s localhost:9200/zipkin-2017-01-18/span/_search -d'{
	"query": {
	"bool": {
	"should": [
	{
	"nested": {
	"path": "annotations",
	"query": {
	"bool": {

	HttpServletRequest httpRequest = (HttpServletRequest) request;
	TraceContextOrSamplingFlags contextOrFlags = contextExtractor.extract(httpRequest);
	Span span = contextOrFlags.context() != null
	? tracer.joinSpan(contextOrFlags.context())
	: tracer.newTrace(contextOrFlags.samplingFlags());

	┌──────────────────────────────┐ ┌──────────────────────────────┐
	│ Tracer: Web │ │ Tracer: App │
	│ │ │ │
	└──────────────────────────────┘ │ └──────────────────────────────┘
	────────────────────────────────────┴───────────────────────────────────

	┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐ ┌─────────────────────────────┐
	(1) │(1) Web tracer starts a trace│
	│ sr │ │ since no headers exist │
	─────────▶ └─────────────────────────────┘

	/**
	* This is an example of a one-way or "messaging span", which is possible by use of the {@link
	* Span#flush()} operator.
	*
	* <p>Note that this uses a span as a kafka key, not because it is recommended, rather as it is
	* convenient for demonstration, since kafka doesn't have message properties.
	*
	* <p>See https://github.com/openzipkin/zipkin/issues/1243
	*/
	public class KafkaExampleIT {