Create a local Kafka cluster for studying and experimentation - It's a Kafka 0.10 cluster but you can install a most recent version changing the lines 12 and 13

README.md

If you are playing around with Kafka you, probably, would prefer to use containers. But, for some reason, you may want to use VMs instead. With this Vagrant script you can get a cluster installed and configured locally with a few steps.

Note that this script doesn't use the most recent version of Kafka. But you can change the version by changing the lines 12 and 13 of the script. Check the Confluent Platform and Apache Kafka Compatibilty table for details.

Instructions to execute and test

1. Install Virtual Box https://www.virtualbox.org/

2. Install Vagrant >= 1.6.4 http://www.vagrantup.com/

3. Copy the Vagrantfile to a folder and run:

$ vagrant up

It will take a couple of minutes.

4. Start Zookeeper

$ vagrant ssh zk
$ sudo zookeeper-server-start /etc/kafka/zookeeper.properties

5. Start the Kafka brokers

$ vagrant ssh kafka101
$ sudo kafka-server-start /etc/kafka/server.properties

Repeat this procedure for kafka102 and kafka103

6. Create a topic

The following commands can be executed inside any VM. Use $ vagrant ssh kafka101 to log in.

$ kafka-topics --zookeeper zk:2181 --create --topic my-topic --partitions 6 --replication-factor 2
$ kafka-topics --zookeeper zk:2181 --describe --topic my-topic

7. Send some messages to the topic using the command line util:

$ kafka-console-producer --broker-list kafka101:9092,kafka102:9092  --topic my-topic

8. Consume the messages

kafka-console-consumer --consumer-property group.id=test-consumer-group --from-beginning --topic my-topic --bootstrap-server kafka101:9092,kafka102:9092

Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  config.vm.box = "ubuntu/trusty64"

  config.vm.provision "shell", inline: <<-SHELL
    sudo add-apt-repository ppa:openjdk-r/ppa
    sudo apt-get update
    sudo apt-get install -y openjdk-8-jdk
    wget -qO - http://packages.confluent.io/deb/3.2/archive.key | sudo apt-key add -
    sudo add-apt-repository "deb [arch=amd64] http://packages.confluent.io/deb/3.2 stable main"
    sudo apt-get update
    sudo apt-get install -y confluent-platform-oss-2.11
  SHELL

  config.vm.define "kafka101" do |kafka101|
    kafka101.vm.hostname = "kafka101"
    kafka101.vm.network "private_network", ip: "172.28.128.3"
    kafka101.vm.provider "virtualbox" do |vb|
      vb.memory = "2048"
    end
    kafka101.vm.provision "shell", inline: <<-SHELL
      sudo -- sh -c -e 'echo "172.28.128.4\tkafka102\n172.28.128.5\tkafka103\n172.28.128.6\tzk"  >> /etc/hosts'
      sudo sed -i 's/broker.id=0/broker.id=101/g' /etc/kafka/server.properties
    SHELL
  end

  config.vm.define "kafka102" do |kafka102|
    kafka102.vm.hostname = "kafka102"
    kafka102.vm.network "private_network", ip: "172.28.128.4"
    kafka102.vm.provider "virtualbox" do |vb|
      vb.memory = "2048"
    end
    kafka102.vm.provision "shell", inline: <<-SHELL
      sudo -- sh -c -e 'echo "172.28.128.3\tkafka101\n172.28.128.5\tkafka103\n172.28.128.6\tzk"  >> /etc/hosts'
      sudo sed -i 's/broker.id=0/broker.id=102/g' /etc/kafka/server.properties
    SHELL
  end

  config.vm.define "kafka103" do |kafka103|
    kafka103.vm.hostname = "kafka103"
    kafka103.vm.network "private_network", ip: "172.28.128.5"
    kafka103.vm.provider "virtualbox" do |vb|
      vb.memory = "2048"
    end
    kafka103.vm.provision "shell", inline: <<-SHELL
      sudo -- sh -c -e 'echo "172.28.128.4\tkafka102\n172.28.128.3\tkafka101\n172.28.128.6\tzk"  >> /etc/hosts'
      sudo sed -i 's/broker.id=0/broker.id=103/g' /etc/kafka/server.properties
    SHELL
  end

  config.vm.define "zk" do |zk|
    zk.vm.hostname = "zk"    
    zk.vm.network "private_network", ip: "172.28.128.6"
    zk.vm.provider "virtualbox" do |vb|
      vb.memory = "1024"
    end
    zk.vm.provision "shell", inline: <<-SHELL
      sudo -- sh -c -e 'echo "172.28.128.3\tkafka101\n172.28.128.4\tkafka102\n172.28.128.5\tkafka103"  >> /etc/hosts'
    SHELL
  end
  
  config.vm.provision "shell", inline: <<-SHELL
    sudo sed -i 's/localhost:2181/zk:2181/g' /etc/kafka/server.properties
  SHELL
end