Install Kafka on OSX via Homebrew

install-kafka.txt

$> brew cask install java

$> brew install kafka

$> vim ~/bin/kafka
# ~/bin/kafka
#!/bin/bash

zkServer start
kafka-server-start.sh /usr/local/etc/kafka/server.properties

$> chmod +x ~/bin/kafka

thank you a lot

After enter VIM I did not get what to do?

~/bin/kafka

#!/bin/bash

zkServer start
kafka-server-start.sh /usr/local/etc/kafka/server.properties

Please anybody can guide to install Kafka.

Homebrew has a services manager and can manage its own Java installation, so an easier way to do this is just

# this will install java 1.8, zookeeper, and kafka
brew install kafka

# this will run ZK and kafka as services
brew services start zookeeper
brew services start kafka

That's it. If you want to stop Kafka, just run the brew services commands in reverse:

brew services stop kafka
brew services stop zookeeper

how to check the port number on which kafka is running ?

@abhaykmr52, the default port for kafka should be 9092. The homebrew uses port 9092. If you are setting up mutliple brokers, you will have multiple server.properties files and each one will specify the port.

How do you install and older version?

Followed this instruction and got an error while trying to publish some messages into topic:

[2018-08-31 19:10:47,854] WARN [Consumer clientId=consumer-1, groupId=console-consumer-42143] Error while fetching metadata with correlation id 32 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

UPDATE.

Looks like default server.properties is not valid for kafka 2.0.0.
It works after adding following lines:

listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092

Thanks this helped!

brew install kafka
sudo mkdir -p /usr/local/var/run/zookeeper/data
sudo chmod 777 /usr/local/var/run/zookeeper/data
zkServer start

mkdir -p /usr/local/var/lib/kafka-logs
sudo chmod 777 /usr/local/var/lib/kafka-logs
/usr/local/Cellar/kafka/2.1.0/libexec/bin/kafka-server-start.sh /usr/local/etc/kafka/server.properties

https://gist.github.com/mayankcpdixit/105d4b15ead728f9d9cacb9af051de48

helpful

Thanks a lot

Great tutorial

On first startup after performing

brew install kafka

and executing in separate terminal sessions

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

and

kafka-server-start /usr/local/etc/kafka/server.properties

Kafka shut down immediately after getting the following error:

ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID Ihfqd5S-Sde60kTsOmcmtA doesn't match stored clusterId Some(NXVqmCxHSMa7Q8OgEaC9IA) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.

So I replaced the stored cluster.id in the file meta.properties located in /usr/local/var/lib/kafka-logs with the required cluster ID from the warning. This did it for me and the kafka server started up without problems.

@rmannhx Thanks for posting

Thanks very much.
Any way to just install the command line tooling (kafka-console-consumer etc) for Kafka but not Kafka 'server and its dependencies' itself.
I only want to the tooling on my machine as Kafka is running elsewhere.
Thanks for all info

sorry anyone knows whats the command to check kafka status using brew
brew services kafka status doesnt work for some reason

Homebrew has a services manager and can manage its own Java installation, so an easier way to do this is just

# this will install java 1.8, zookeeper, and kafka
brew install kafka

# this will run ZK and kafka as services
brew services start zookeeper
brew services start kafka

That's it. If you want to stop Kafka, just run the brew services commands in reverse:

brew services stop kafka
brew services stop zookeeper

sory mate
does this also start schema registry and everything else?
is this command enough to be able to work with kafka fully?

Homebrew has a services manager and can manage its own Java installation, so an easier way to do this is just

# this will install java 1.8, zookeeper, and kafka
brew install kafka

# this will run ZK and kafka as services
brew services start zookeeper
brew services start kafka

That's it. If you want to stop Kafka, just run the brew services commands in reverse:

brew services stop kafka
brew services stop zookeeper

THANK YOU!

If you want to use kraft instead of zookeeper, just replace the default server.properties with the default kraft server.properties.

You can use /usr/local/ instead of $(brew --prefix) if you prefer.

brew install kafka

mv $(brew --prefix)/etc/kafka/kraft/server.properties $(brew --prefix)/etc/kafka/server.properties
kafka-storage format -t $(kafka-storage random-uuid) -c $(brew --prefix)/etc/kafka/server.properties

Then the zookeeper command is not necessary:

brew services start kafka
brew services info kafka
brew services stop kafka

@spartiv -- the info is for showing the status

---

After installing java (via kafka) I told OSX to use the java that I installed: (not sure if this is necessary)

sudo ln -sfn $(brew --prefix)/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk

WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient) whats the meaning of this ?