Connect to kafka from a pods in kubernetes clusters

kafka_kubernetes_pod.md

If you want to integrate kubernetes with legacy system which are not running on kubernetes. You can connect from a kubernetes pods with those services by docker interfaces. The kafka is running on the host network at port 9092.

We have kafka client

import logging

# for checking internal connection of kafka module
logging.basicConfig(level=logging.DEBUG)
LOG = logging.getLogger('test.kafka')

from kafka import KafkaConsumer, KafkaProducer
consumer = KafkaConsumer('hello', bootstrap_servers=["127.0.0.1","172.17.0.1"])
for message in consumer:
    try: 
        LOG.info("Recv message: %s", message)
    except Exception as e:  
        pass

and our pod configuration test_kafka.yaml

apiVersion: v1
kind: Pod
metadata:
  name: kafka-test
  namespace: default
spec:
  containers:
  - name: kafka-test
    image: <private_registry>/kafka-test
    imagePullPolicy: Always
  restartPolicy: OnFailure
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet

Just apply it: kubectl apply -f test_kafka.yaml