Example of kafka-python producer using Avro

README.md

Notes:

• This uses the python loremipsum generator. You can replace g.generate_sentence() with your own string or generator of choice
• Make sure to replace schema_path in the script with your own schema path
• No guarantees that this is the best way to loop over records and publish them to Kafka, just a demo. Before adapting this script, take a look at the Avro documentation...the writer might not need to be instantiated for every record, for example. I'm just not sure.

Instructions

1. Create a Kafka topic to test this on (default in the script is 'test')
2. Replace schema_path with the path to user.avsc
3. Open up a console Kafka consumer (see the 'quick start' section in Kafka's documentation)
4. From the command line, python kafka_avro_python_example.py
5. Because the records are Avro-encoded, you'll see some funky characters in the console consumer. This is expected.

kafka_avro_python_example.py

import io
import avro.schema
import avro.io
import lipsum
import random

from kafka.client import KafkaClient
from kafka.producer import SimpleProducer, KeyedProducer

g = lipsum.Generator()
kafka = KafkaClient("localhost:9092")
producer = SimpleProducer(kafka)

# Path to user.avsc avro schema
schema_path="/Your/schema/path/user.avsc"

# Kafka topic
topic = "test"

schema = avro.schema.parse(open(schema_path).read())


for i in xrange(2000):

	writer = avro.io.DatumWriter(schema)
	bytes_writer = io.BytesIO()
	encoder = avro.io.BinaryEncoder(bytes_writer)
	writer.write({"name": g.generate_sentence(), "favorite_color": g.generate_sentence(), "favorite_number": random.randint(0,10)}, encoder)
	raw_bytes = bytes_writer.getvalue()
	producer.send_messages(topic, raw_bytes)

user.avsc

{"namespace": "example.avro",
 "type": "record",
 "name": "User",
 "fields": [
     {"name": "name", "type": "string"},
     {"name": "favorite_number",  "type": ["int", "null"]},
     {"name": "favorite_color", "type": ["string", "null"]}
 ]
}

how to tell which partition your data is sent to?