protobuf.md

Install protobuf compiler

brew install protobuf

protoc compiler is responsible to generate language-specific stubs based on the protobuf schema.

Stubs are necessary to encode and decode data. Therefore, encoding always requires language runtime environment and protobuf schema. Stubs implement the Protobuf API to read messages and write messages.

To use the Protobuf API you need the language-specific library. For Python, this is protobuf library.

pip install protobuf

Now you can generate the stub in Python. This will crete foo_pb2.py.

protoc --python-out=. foo.proto

Inspect a protobuf message

$ protoc --decode foo.BarMsg foo.proto < foo.message

JSON-protobuf bridge

To convert a json file into protobuf message, you need the proto file and related stub.

cat sample.json | pbgen > sample.message

Convert json to protobuf message and publish to kafka

cat sample.json | pbgen | kafkacat -Pt foo

process substitution

kafkacat -Plt us-east-01.cdp.queue.bcss-request.mx.low-priority < <(cat braze_sync_request.json | ./pbgen)

kafkacat -Pt us-east-01.cdp.queue.bcss-request.mx.low-priority braze_sync_request.message

Consume from kafka and convert to json

kafkacat -C -u -q -f "%R%s" -t foo7 | pq --protofile todolist.proto --msgtype protoblog.TodoList --stream i32be

PQ

Install pq

cargo install pq

Convert protobuf message to json

cat foo.message | pq --protofile foo.proto --msgtype foo.BarMsg

Questions

• what is proto extentsion?
• what is compiled file?

Next

https://developers.google.com/protocol-buffers/docs/pythontutorial
https://lecstor.com/kafka-cheatsheet/