Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Separates Offsets from different context (Spark vs PSL) #404

Merged
merged 5 commits into from Dec 9, 2020

Conversation

jiangmichaellll
Copy link
Contributor

Spark offset means msgs with offsets less than or equal to have been processed, and next to be delivered is spark_offset+1.
PSL offset means msgs with offsets less than have been processed, and next to be delivered is psl_offset.

Spark side:
SparkSourceOffset, SparkPartitionOffset
PSL side:
PslSourceOffset, PslPartitionOffset

Also added utils method to convert between PSL and Spark.

@jiangmichaellll jiangmichaellll requested a review from a team as a code owner December 4, 2020 04:24
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Dec 4, 2020
@jiangmichaellll jiangmichaellll self-assigned this Dec 4, 2020
@product-auto-label product-auto-label bot added the api: pubsublite Issues related to the googleapis/java-pubsublite API. label Dec 4, 2020
@codecov
Copy link

codecov bot commented Dec 4, 2020

Codecov Report

Merging #404 (7356b05) into master (8f26f54) will increase coverage by 0.09%.
The diff coverage is 86.58%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #404      +/-   ##
============================================
+ Coverage     71.85%   71.95%   +0.09%     
- Complexity      828      833       +5     
============================================
  Files           156      158       +2     
  Lines          4356     4457     +101     
  Branches        221      223       +2     
============================================
+ Hits           3130     3207      +77     
- Misses         1106     1125      +19     
- Partials        120      125       +5     
Impacted Files Coverage Δ Complexity Δ
...google/cloud/pubsublite/spark/PslSourceOffset.java 76.92% <72.72%> (+0.25%) 5.00 <5.00> (-6.00) ⬆️
...ogle/cloud/pubsublite/spark/SparkSourceOffset.java 85.10% <85.10%> (ø) 17.00 <17.00> (?)
...m/google/cloud/pubsublite/spark/PslSparkUtils.java 59.37% <95.00%> (+59.37%) 6.00 <5.00> (+6.00)
...gle/cloud/pubsublite/spark/PslPartitionOffset.java 100.00% <100.00%> (ø) 2.00 <1.00> (ø)
...e/cloud/pubsublite/spark/SparkPartitionOffset.java 100.00% <100.00%> (ø) 2.00 <2.00> (?)
...e/v1/stub/GrpcPublisherServiceCallableFactory.java 25.00% <0.00%> (-25.00%) 2.00% <0.00%> (-2.00%)
.../v1/stub/GrpcSubscriberServiceCallableFactory.java 25.00% <0.00%> (-25.00%) 2.00% <0.00%> (-2.00%)
...GrpcPartitionAssignmentServiceCallableFactory.java 25.00% <0.00%> (-25.00%) 2.00% <0.00%> (-2.00%)
...google/cloud/pubsublite/v1/AdminServiceClient.java 74.13% <0.00%> (-17.25%) 45.00% <0.00%> (-10.00%)
.../v1/stub/GrpcTopicStatsServiceCallableFactory.java 25.00% <0.00%> (-12.50%) 2.00% <0.00%> (-1.00%)
... and 47 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8f26f54...8e4b6d5. Read the comment docs.

@dpcollins-google
Copy link
Collaborator

LGTM other than don't write your own equals or hashcode methods. Ping me when you've fixed that issue

@jiangmichaellll jiangmichaellll merged commit b745f58 into master Dec 9, 2020
@jiangmichaellll jiangmichaellll deleted the jiangmichael-spark-offsets branch December 9, 2020 06:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: pubsublite Issues related to the googleapis/java-pubsublite API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants