Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Use TopicStats to implement getSplitBacklog #228

Merged
merged 4 commits into from Sep 15, 2020

Conversation

palmere-google
Copy link
Contributor

We use the TopicStatsService to compute the backlog size for the
assigned splits in the PubsubLiteUnbounded Reader.

We will refresh backlog information every ten seconds, and will use
stale information for up to a minute if the topic stats serivce is
unavailable.

Constructing the TopicBacklogReader in SubscriberOptions is a bit
awkward, but I wanted to make sure we don't need to re-resolve the
subscription -> topic mapping when we split or merge the source.
Otherwise we end up with a hard dependency on the admin service.

@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Aug 28, 2020
@codecov
Copy link

codecov bot commented Aug 28, 2020

Codecov Report

Merging #228 into master will decrease coverage by 0.47%.
The diff coverage is 43.75%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #228      +/-   ##
============================================
- Coverage     72.71%   72.23%   -0.48%     
+ Complexity      752      746       -6     
============================================
  Files           139      135       -4     
  Lines          3987     4009      +22     
  Branches        209      207       -2     
============================================
- Hits           2899     2896       -3     
- Misses          965      991      +26     
+ Partials        123      122       -1     
Impacted Files Coverage Δ Complexity Δ
...oud/pubsublite/beam/PubsubLiteUnboundedSource.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...oogle/cloud/pubsublite/beam/SubscriberOptions.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...oud/pubsublite/beam/PubsubLiteUnboundedReader.java 72.95% <91.30%> (+4.26%) 24.00 <4.00> (+4.00)
.../java/com/google/cloud/pubsublite/ProjectPath.java 0.00% <0.00%> (-60.00%) 0.00% <0.00%> (-4.00%)
...n/java/com/google/cloud/pubsublite/TopicPaths.java 77.41% <0.00%> (-22.59%) 7.00% <0.00%> (+5.00%) ⬇️
...in/java/com/google/cloud/pubsublite/TopicPath.java 66.66% <0.00%> (-20.84%) 2.00% <0.00%> (-2.00%)
...java/com/google/cloud/pubsublite/LocationPath.java 66.66% <0.00%> (-20.84%) 2.00% <0.00%> (-2.00%)
.../com/google/cloud/pubsublite/SubscriptionPath.java 66.66% <0.00%> (-20.84%) 2.00% <0.00%> (-2.00%)
...com/google/cloud/pubsublite/SubscriptionPaths.java 90.90% <0.00%> (-9.10%) 8.00% <0.00%> (+6.00%) ⬇️
...in/java/com/google/cloud/pubsublite/TopicName.java 66.66% <0.00%> (-8.34%) 2.00% <0.00%> (-1.00%)
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e30896e...635b040. Read the comment docs.

@product-auto-label product-auto-label bot added the api: pubsublite Issues related to the googleapis/java-pubsublite API. label Aug 29, 2020
Copy link
Collaborator

@dpcollins-google dpcollins-google left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll also have to integrate the changes made to the topic path creation logic in this pr.

We use the TopicStatsService to compute the backlog size for the
assigned splits in the PubsubLiteUnbounded Reader.

We will refresh backlog information every ten seconds, and will use
stale information for up to a minute if the topic stats serivce is
unavailable.

Constructing the TopicBacklogReader in SubscriberOptions is a bit
awkward, but I wanted to make sure we don't need to re-resolve the
subscription -> topic mapping when we split or merge the source.
Otherwise we end up with a hard dependency on the admin service.
@palmere-google
Copy link
Contributor Author

Okay, rebased to include the path changes

PTAL

@palmere-google palmere-google force-pushed the palmere_beam branch 2 times, most recently from 19d237b to 8a9ad0b Compare September 3, 2020 18:16
@palmere-google palmere-google added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 3, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Sep 3, 2020
@palmere-google palmere-google merged commit 9a889a9 into master Sep 15, 2020
@palmere-google palmere-google deleted the palmere_beam branch September 15, 2020 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: pubsublite Issues related to the googleapis/java-pubsublite API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants