Skip to content

Instantly share code, notes, and snippets.

@denzhel

denzhel/kafka_connect_partition_assignment.md

Created July 14, 2022 12:36
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save denzhel/73ab9fe2afbc736896d3e41ae7f3f465 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save denzhel/73ab9fe2afbc736896d3e41ae7f3f465 to your computer and use it in GitHub Desktop.
Download ZIP
kafka connect assign partitions to tasks better
Raw
kafka_connect_partition_assignment.md

We have 1000 topics, 1 partition each. Our kafka connect has the following config:

tasks.max: 10

We expected each each task to handle 100 topics but NO ! Instead, single topics partition will be handled by the first task only. Meaning, the first task will handle ALL the 1000 topics. Why ? RangeAssignor is the default kafka connect partition assingor.

To allow a better even spread across all the workers, I configured our kafka connect with:

"consumer.override.partition.assignment.strategy": "org.apache.kafka.clients.consumer.RoundRobinAssignor"

Once I did that, all workers started handling topics.

Issue: https://issues.apache.org/jira/browse/KAFKA-13136

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment