New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(pubsublite): routing publisher implementation #3277
Conversation
Handles increasing topic partition count.
WANT_LGTM=@manuelmenzella-google, @palmere-google If anyone feels strongly about waiting for the new partition publishers to successfully start before routing messages to them, feel free to let me know. It can be done, but introduces some complexity, so I went with this. If any partition publisher fails to start up, the entire routing publisher will be terminated anyway. |
8a2e7b9
to
c2dea0a
Compare
…nitiateShutdown since it never varies
b2ae47b
to
a2fd97e
Compare
partitionCountWatcher
polls for updates to the number of topic partitions.routingPublisher
routes messages to multiplesinglePartitionPublishers
and handles the topic partition count increasing. Decreases will terminate the routingPublisher.The initial set of partition publishers must successfully start before the routingPublisher accepts published messages. However, when new partition publishers are added later, messages may be routed to them while they are starting up. The messages are queued in the
publishMessageBatcher
and sent to the server when the Publish stream connects.The entire
routingPublisher
tree terminates if thepartitionCountWatcher
or any of thesinglePartitionPublishers
terminate.