TraceView on Docker

gistfile1.md

To get traces generated from a container arrive at the docker host I had to do the following:

On the docker host I run the following to jiggy wig the tracelyzer to listen on something other than 127.0.0.1:

nc -l 172.17.42.1 7831 | nc 127.0.0.1 7831

update: This doesn't forward UDP packets correctly to the dashboard because of a netcat issue. (it was a temporary hack anyways) An IPTables rule or another alternative (socat?) would be a better solution until the Tracelyzer bugfix is shipped.

update 2: Per Adam Tucker, use http://brokestream.com/udp_redirect.c instead

I used a pre-existing container with Ruby installed (this gets me into a shell in the container):

docker run -i -t paintedfox/ruby /bin/bash

..then ran the following to set the container up:

apt-get -y install git wget
wget https://www.tracelytics.com/install_tracelytics.sh 
sh ./install_tracelytics.sh <my UID>
gem install oboe pry

...note that I am installing the tracelyzer into the container so it doesn't complain about the /var/lib/tracelyzer/settings file. I immediately turn it off:

/etc/init.d/tracelyzer stop

update: /var/lib/tracelyzer/settings can be shared from the docker host into the container by using the docker run -v option.

On the docker host (not in the container) I also stop tracelyzer and instead run

/usr/bin/tracelyzer -r

...to manually watch traces generated in the console.

Then from the container shell I boot up pry (a ruby console) and run the following

require 'oboe'
Oboe::Config[:reporter_host] = "172.17.42.1"
Oboe::Config[:verbose] = true
Oboe::Reporter.start
Oboe::API::report_init('ruby')
Oboe::Config[:tracing_mode] = "always"
Oboe::Config[:sample_rate] = 1e6
Oboe::API.start_trace('rack', nil, { :some_key => true }) do sleep 1 end

To which I get trace output on the docker host similar to the following:

root@ip-10-183-149-249:~# /usr/bin/tracelyzer -r
Debug mode enabled; ready to receive Tracelytics reports.
Report 1 (339 bytes)
	X-Trace-Header : 2 	 X-Trace Report ver 1.1
	X-Trace : 2 	 1BBFFBE97667B26DAEE69209FA42E6D9E06E2FDC1B985A8C51B7C8C838
	Layer : 2 	 rack
	Label : 2 	 entry
	__Init : 2 	 1
	Force : 2 	 true
	Ruby.Platform.Version : 2 	 x86_64-linux
	Ruby.Version : 2 	 2.0.0
	Ruby.Oboe.Version : 2 	 2.3.4.1
	TID : 18 	 3665
	Timestamp_u : 18 	 1389209506808111
	Hostname : 2 	 ce49cf8fc4da
Report 2 (240 bytes)
	X-Trace-Header : 2 	 X-Trace Report ver 1.1
	X-Trace : 2 	 1BBFFBE97667B26DAEE69209FA42E6D9E06E2FDC1B2E1430BB7DEB6923
	Edge : 2 	 985A8C51B7C8C838
	Layer : 2 	 rack
	Label : 2 	 exit
	TID : 18 	 3665
	Timestamp_u : 18 	 1389209506808549
	Hostname : 2 	 ce49cf8fc4da
Report 3 (233 bytes)
	X-Trace-Header : 2 	 X-Trace Report ver 1.1
	X-Trace : 2 	 1BD39BB947703CF3D22B252DA6113D0567F05CDD9556190C52164CD2C9
	Layer : 2 	 rack
	Label : 2 	 entry
	some_key : 2 	 true
	TID : 18 	 3677
	Timestamp_u : 18 	 1389210218691586
	Hostname : 2 	 ce49cf8fc4da
Report 4 (240 bytes)
	X-Trace-Header : 2 	 X-Trace Report ver 1.1
	X-Trace : 2 	 1BD39BB947703CF3D22B252DA6113D0567F05CDD952067016A36E8CE14
	Edge : 2 	 56190C52164CD2C9
	Layer : 2 	 rack
	Label : 2 	 exit
	TID : 18 	 3677
	Timestamp_u : 18 	 1389210219692102
	Hostname : 2 	 ce49cf8fc4da

When this was written, tracelyzer used to be hard-coded to 127.0.0.1. Since then, I believe the tracelyzer/liboboe now supports a configurable bind address.

Assuming this is true, then the netcat/udp_redir step is no longer needed.