Install Kafka on OSX via Homebrew

install-kafka.txt

$> brew cask install java

$> brew install kafka

$> vim ~/bin/kafka
# ~/bin/kafka
#!/bin/bash

zkServer start
kafka-server-start.sh /usr/local/etc/kafka/server.properties

$> chmod +x ~/bin/kafka

Thanks this helped!

brew install kafka
sudo mkdir -p /usr/local/var/run/zookeeper/data
sudo chmod 777 /usr/local/var/run/zookeeper/data
zkServer start

mkdir -p /usr/local/var/lib/kafka-logs
sudo chmod 777 /usr/local/var/lib/kafka-logs
/usr/local/Cellar/kafka/2.1.0/libexec/bin/kafka-server-start.sh /usr/local/etc/kafka/server.properties

https://gist.github.com/mayankcpdixit/105d4b15ead728f9d9cacb9af051de48

helpful

Thanks a lot

Great tutorial

On first startup after performing

brew install kafka

and executing in separate terminal sessions

zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties

and

kafka-server-start /usr/local/etc/kafka/server.properties

Kafka shut down immediately after getting the following error:

ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID Ihfqd5S-Sde60kTsOmcmtA doesn't match stored clusterId Some(NXVqmCxHSMa7Q8OgEaC9IA) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.

So I replaced the stored cluster.id in the file meta.properties located in /usr/local/var/lib/kafka-logs with the required cluster ID from the warning. This did it for me and the kafka server started up without problems.

@rmannhx Thanks for posting

Thanks very much.
Any way to just install the command line tooling (kafka-console-consumer etc) for Kafka but not Kafka 'server and its dependencies' itself.
I only want to the tooling on my machine as Kafka is running elsewhere.
Thanks for all info

sorry anyone knows whats the command to check kafka status using brew
brew services kafka status doesnt work for some reason

Homebrew has a services manager and can manage its own Java installation, so an easier way to do this is just

# this will install java 1.8, zookeeper, and kafka
brew install kafka

# this will run ZK and kafka as services
brew services start zookeeper
brew services start kafka

That's it. If you want to stop Kafka, just run the brew services commands in reverse:

brew services stop kafka
brew services stop zookeeper

sory mate
does this also start schema registry and everything else?
is this command enough to be able to work with kafka fully?

Homebrew has a services manager and can manage its own Java installation, so an easier way to do this is just

# this will install java 1.8, zookeeper, and kafka
brew install kafka

# this will run ZK and kafka as services
brew services start zookeeper
brew services start kafka

That's it. If you want to stop Kafka, just run the brew services commands in reverse:

brew services stop kafka
brew services stop zookeeper

THANK YOU!

If you want to use kraft instead of zookeeper, just replace the default server.properties with the default kraft server.properties.

You can use /usr/local/ instead of $(brew --prefix) if you prefer.

brew install kafka

mv $(brew --prefix)/etc/kafka/kraft/server.properties $(brew --prefix)/etc/kafka/server.properties
kafka-storage format -t $(kafka-storage random-uuid) -c $(brew --prefix)/etc/kafka/server.properties

Then the zookeeper command is not necessary:

brew services start kafka
brew services info kafka
brew services stop kafka

@spartiv -- the info is for showing the status

---

After installing java (via kafka) I told OSX to use the java that I installed: (not sure if this is necessary)

sudo ln -sfn $(brew --prefix)/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk

WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient) whats the meaning of this ?