Skip to content

Instantly share code, notes, and snippets.

@StevenACoffman
Created September 26, 2017 14:36
Show Gist options
  • Save StevenACoffman/c1a3797ecc1b9fb40b497c08cb938334 to your computer and use it in GitHub Desktop.
Save StevenACoffman/c1a3797ecc1b9fb40b497c08cb938334 to your computer and use it in GitHub Desktop.
Fluent kafka
Q: My structured log messages contain a field (destinations) that described one or more kafka topics that the messages should be produced to.
Is there a recommended way (or plugin) to fan-out to multiple destinations?
A: use record_reformer to add field to tag, then use tag routing system
Q: How do you split an array (comma separated list) when you don't know how many items will be in it?
A: If you parse such array value in input phase, `types` parameter may help. https://docs.fluentd.org/v0.12/articles/parser_regexp#types
Q: Can the topic key be a comma separated list of topics so that it produces to multiple topics?
https://github.com/fluent/fluent-plugin-kafka
You can use the tag as topic to create multiple topics per data stream
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment