Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(pubsublite): routing publisher implementation #3277

Merged
merged 8 commits into from Dec 2, 2020

Conversation

tmdiep
Copy link
Contributor

@tmdiep tmdiep commented Nov 25, 2020

partitionCountWatcher polls for updates to the number of topic partitions.

routingPublisher routes messages to multiple singlePartitionPublishers and handles the topic partition count increasing. Decreases will terminate the routingPublisher.

The initial set of partition publishers must successfully start before the routingPublisher accepts published messages. However, when new partition publishers are added later, messages may be routed to them while they are starting up. The messages are queued in the publishMessageBatcher and sent to the server when the Publish stream connects.

The entire routingPublisher tree terminates if the partitionCountWatcher or any of the singlePartitionPublishers terminate.

Handles increasing topic partition count.
@tmdiep tmdiep requested a review from a team as a code owner November 25, 2020 11:40
@product-auto-label product-auto-label bot added the api: pubsublite Issues related to the Pub/Sub Lite API. label Nov 25, 2020
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Nov 25, 2020
@tmdiep
Copy link
Contributor Author

tmdiep commented Nov 25, 2020

WANT_LGTM=@manuelmenzella-google, @palmere-google
Other reviewers optional, but comments welcome.

If anyone feels strongly about waiting for the new partition publishers to successfully start before routing messages to them, feel free to let me know. It can be done, but introduces some complexity, so I went with this. If any partition publisher fails to start up, the entire routing publisher will be terminated anyway.

@tmdiep tmdiep merged commit 88e5466 into googleapis:master Dec 2, 2020
@tmdiep tmdiep deleted the routing_publisher branch December 2, 2020 03:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: pubsublite Issues related to the Pub/Sub Lite API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants