Last active
March 6, 2024 21:58
-
-
Save rushi47/823e97a4793027ee9a05b9c16a58b05b to your computer and use it in GitHub Desktop.
Dynamic topic for output kafka in fluentbit
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Place below config in fluent bit.config. And annotate your pods with `k8s_annotations_fluentbit.topic : <destination-topic>`. | |
For the pods which have this annotation defined, kafka will output : respective `<destination-topic>` | |
[FILTER] | |
Name kubernetes | |
Match kube.* | |
Merge_Log On | |
Keep_Log Off | |
K8S-Logging.Parser On | |
K8S-Logging.Exclude On | |
[FILTER] | |
Name nest | |
Match * | |
Operation lift | |
Nested_under kubernetes | |
Add_prefix kubernetes_ | |
[FILTER] | |
Name nest | |
Match * | |
Operation lift | |
Nested_under kubernetes_annotations | |
Add_prefix k8s_annotations_ | |
[FILTER] | |
Name nest | |
Match * | |
Operation nest | |
Wildcard kubernetes_* | |
Nest_under kubernetes | |
[OUTPUT] | |
Name kafka | |
Match kube.* | |
Brokers ${KAFKA_BROKER} | |
Topics default_topic_here | |
Topic_key k8s_annotations_fluentbit.topic | |
Dynamic_topic On | |
...remaining config |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment