New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Use TopicStats to implement getSplitBacklog #228
Conversation
e75f6e0
to
fe55c09
Compare
Codecov Report
@@ Coverage Diff @@
## master #228 +/- ##
============================================
- Coverage 72.71% 72.23% -0.48%
+ Complexity 752 746 -6
============================================
Files 139 135 -4
Lines 3987 4009 +22
Branches 209 207 -2
============================================
- Hits 2899 2896 -3
- Misses 965 991 +26
+ Partials 123 122 -1 Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You'll also have to integrate the changes made to the topic path creation logic in this pr.
...ublite-beam-io/src/main/java/com/google/cloud/pubsublite/beam/PubsubLiteUnboundedReader.java
Show resolved
Hide resolved
...ublite-beam-io/src/main/java/com/google/cloud/pubsublite/beam/PubsubLiteUnboundedReader.java
Show resolved
Hide resolved
...ublite-beam-io/src/main/java/com/google/cloud/pubsublite/beam/PubsubLiteUnboundedReader.java
Outdated
Show resolved
Hide resolved
...ublite-beam-io/src/main/java/com/google/cloud/pubsublite/beam/PubsubLiteUnboundedReader.java
Outdated
Show resolved
Hide resolved
pubsublite-beam-io/src/main/java/com/google/cloud/pubsublite/beam/SubscriberOptions.java
Outdated
Show resolved
Hide resolved
pubsublite-beam-io/src/main/java/com/google/cloud/pubsublite/beam/SubscriberOptions.java
Outdated
Show resolved
Hide resolved
We use the TopicStatsService to compute the backlog size for the assigned splits in the PubsubLiteUnbounded Reader. We will refresh backlog information every ten seconds, and will use stale information for up to a minute if the topic stats serivce is unavailable. Constructing the TopicBacklogReader in SubscriberOptions is a bit awkward, but I wanted to make sure we don't need to re-resolve the subscription -> topic mapping when we split or merge the source. Otherwise we end up with a hard dependency on the admin service.
5b7c5eb
to
635b040
Compare
Okay, rebased to include the path changes PTAL |
pubsublite-beam-io/src/main/java/com/google/cloud/pubsublite/beam/SubscriberOptions.java
Show resolved
Hide resolved
19d237b
to
8a9ad0b
Compare
8a9ad0b
to
aceb092
Compare
We use the TopicStatsService to compute the backlog size for the
assigned splits in the PubsubLiteUnbounded Reader.
We will refresh backlog information every ten seconds, and will use
stale information for up to a minute if the topic stats serivce is
unavailable.
Constructing the TopicBacklogReader in SubscriberOptions is a bit
awkward, but I wanted to make sure we don't need to re-resolve the
subscription -> topic mapping when we split or merge the source.
Otherwise we end up with a hard dependency on the admin service.