Zipkin with Kafka

docker-compose.yml

kafka:
  image: spotify/kafka
  environment:
    - ADVERTISED_HOST
    - ADVERTISED_PORT=9092
  ports:
    - 2181:2181
    - 9092:9092

cassandra:
  image: openzipkin/zipkin-cassandra:1.25.0
  ports:
    - 9042:9042

# The collector process is used for legacy zipkin instrumentation, which log via
# scribe. It can also poll kafka, when the KAFKA_ZOOKEEPER variable is set.
collector:
  image: openzipkin/zipkin-collector:1.25.0
  environment:
    - KAFKA_ZOOKEEPER
    - TRANSPORT_TYPE=kafka
    - STORAGE_TYPE=cassandra
  expose:
    - 9410
  ports:
    - 9410:9410
    - 9900:9900
  links:
    - kafka:kafka
    - cassandra:storage

# The query process services the UI, and also exposes a POST endpoint that
# instrumentation can send trace data to.
query:
  image: openzipkin/zipkin-query:1.25.0
  environment:
    # Remove TRANSPORT_TYPE to disable tracing
    - TRANSPORT_TYPE=http
    - STORAGE_TYPE=cassandra
  expose:
    - 9411
  ports:
    - 9411:9411
    - 9901:9901
  links:
    - cassandra:storage

web:
  image: openzipkin/zipkin-web:1.25.0
  environment:
    # Remove TRANSPORT_TYPE to disable tracing
    - TRANSPORT_TYPE=http
  ports:
    - 8080:8080
    - 9990:9990
  links:
    - query

setup.md

To get things working locally with docker compose I used the below process:

1. Setup the environment variables used by compose

The reason we need this is because we need kafka to advertise an address that can be reached by both our local machine (mac) and from within the collector docker container. There may be a more elegant way to do this but this is what I have so far.

export ADVERTISED_HOST="$(docker-machine ip `docker-machine active`)"
export KAFKA_ZOOKEEPER=$ADVERTISED_HOST

2. Begin docker compose

Make sure to use the compose file in this gist because it sets up kafka differently.

docker-compose up

3. Create the zipkin topic before sending any traffic to it

Because we only have one node we need to make sure we setup the topic so that it doesn't try to replicate. Or at least I think that is why it fails when you don't do this. I run this right after the kafka container is working and before any requests are sent to kafka. Once a request is sent to kafka this will no longer work since the topic will be created with default settings that will not work properly.

This is using a shell script that comes with a kafka download tested with 0.8.x but may work for 0.9.x http://kafka.apache.org/downloads.html

kafka-topics.sh --create --zookeeper ${ADVERTISED_HOST}:2181 --replication-factor 1 --partitions 1 --topic zipkin

4. Send traffic to kafka

At this point the test you posted in the ticket should work as expected.

Also you can use the following scripts to test the ability to send and recieve traffic locally if you need to debug the kafka setup:

kafka-topics.sh --list --zookeeper ${ADVERTISED_HOST}:2181
kafka-console-producer.sh --broker-list ${ADVERTISED_HOST}:9092 --topic zipkin
kafka-console-consumer.sh --zookeeper ${ADVERTISED_HOST}:2181 --topic zipkin