Rule engine: Kafka consumer group per partition

[smatvienko-tb]

Rule engine: Kafka consumer group per partition

Many rule engine's “.topic.” served under a single consumer group:
re-LowPriority-consumer
tb_rule_engine.LowPriority.1
tb_rule_engine.LowPriority.2
tb_rule_engine.LowPriority.3

The problem: one fail - all rebalancing
The solution: use consumer group per topic.partition

Hint: Regarding Kafka partition, we always use a single partition 0. ThingsBoard partition are simple the part a topic name like “*.topic.number”

Tests passed:

General checklist

• You have reviewed the guidelines document.
• Labels that classify your pull request have been added.
• The milestone is specified and corresponds to fix version.
• Description references specific issue.
• Description contains human-readable scope of changes.
• Description contains brief notes about what needs to be added to the documentation.
• No merge conflicts, commented blocks of code, code formatting issues.
• Changes are backward compatible or upgrade script is provided.
• Similar PR is opened for PE version to simplify merge. Crosslinks between PRs added. Required for internal contributors only.

Front-End feature checklist

• Screenshots with affected component(s) are added. The best option is to provide 2 screens: before and after changes;
• If you change the widget or other API, ensure it is backward-compatible or upgrade script is present.
• Ensure new API is documented here

Back-End feature checklist

• Added corresponding unit and/or integration test(s). Provide written explanation in the PR description if you have failed to add tests.
• If new dependency was added: the dependency tree is checked for conflicts.
• If new service was added: the service is marked with corresponding @TbCoreComponent, @TbRuleEngineComponent, @TbTransportComponent, etc.
• If new REST API was added: the RestClient.java was updated, issue for Python REST client is created.
• If new yml property was added: make sure a description is added (above or near the property).

[ashvayka]

Is there any proof that this is solving the problem of re-balancing performance?

[smatvienko-tb]

Is there any proof that this is solving the problem of re-balancing performance?

Kafka consumer groups perform a rebalancing process each time a consumer joins (subscribes) or leaves (unsubscribes) the group. This is necessary to ensure that the partitions of the topics being consumed are evenly distributed among the available consumers in the group.

Here’s a brief overview of how rebalancing works in a Kafka consumer group:

1. Consumer Joins or Leaves: When a new consumer joins a consumer group or an existing consumer leaves the group (due to failure or shutdown), the group coordinator (a broker responsible for managing consumer groups) detects this change.

2. Rebalance Initiation: The group coordinator triggers a rebalance operation. During rebalancing, all consumers in the group stop consuming messages.

3. Partition Reassignment: The coordinator assigns partitions to the consumers based on the partition assignment strategy configured (like round-robin, range, or sticky). This strategy determines how partitions are distributed among the consumers in the group.

4. Partition Consumption: After the partitions are reassigned, consumers start consuming messages from their newly assigned partitions.

5. Continued Monitoring: The coordinator continues to monitor the group for any further changes in consumer membership or failures.

About the ThingsBoard-specific usage of Kafka

Give: Each consumer in the group subscribes to a different topic, and there is no overlap in the topics they consume.

Even in this scenario, where each consumer is consuming a separate topic, Kafka's consumer group coordination doesn't inherently recognize or optimize for the fact that there is no direct competition for partitions among consumers. The group coordinator does not have the logic to bypass rebalancing when each consumer subscribes to a unique topic. The fundamental behavior of triggering a rebalance on join or leave events is still followed.