New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Separates Offsets from different context (Spark vs PSL) #404
Conversation
Codecov Report
@@ Coverage Diff @@
## master #404 +/- ##
============================================
+ Coverage 71.85% 71.95% +0.09%
- Complexity 828 833 +5
============================================
Files 156 158 +2
Lines 4356 4457 +101
Branches 221 223 +2
============================================
+ Hits 3130 3207 +77
- Misses 1106 1125 +19
- Partials 120 125 +5 Continue to review full report at Codecov.
|
...e-spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/SparkSourceOffset.java
Show resolved
Hide resolved
...ite-spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslSourceOffset.java
Outdated
Show resolved
Hide resolved
...e-spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/SparkSourceOffset.java
Show resolved
Hide resolved
LGTM other than don't write your own equals or hashcode methods. Ping me when you've fixed that issue |
Spark offset means msgs with offsets less than or equal to have been processed, and next to be delivered is spark_offset+1.
PSL offset means msgs with offsets less than have been processed, and next to be delivered is psl_offset.
Spark side:
SparkSourceOffset, SparkPartitionOffset
PSL side:
PslSourceOffset, PslPartitionOffset
Also added utils method to convert between PSL and Spark.