Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Spark micro batch processing #426

Merged
merged 48 commits into from Dec 16, 2020
Merged

feat: Spark micro batch processing #426

merged 48 commits into from Dec 16, 2020

Conversation

jiangmichaellll
Copy link
Contributor

@jiangmichaellll jiangmichaellll commented Dec 14, 2020 •
edited

Implementation of spark micro batch processing. Many parts are similar to continuous processing.

@jiangmichaellll
PSL spark sql streaming utility classes
d88265c
@jiangmichaellll
update deps
ff546de
@jiangmichaellll
addressed comments
65dbca4
@jiangmichaellll
add back ignoreclasses
903fa6a
@jiangmichaellll
add used undeclared deps
cdab064
@jiangmichaellll
continuous streaming main things
b42e85e
@jiangmichaellll
address comments
0d7e816
@jiangmichaellll
Merge branch 'jiangmichael-spark-utils' into jiangmichael-spark-conti…
1d6d4ba 
…nuous-processing
@jiangmichaellll
address comments
356bf0b
@jiangmichaellll
Merge branch 'jiangmichael-spark-utils' into jiangmichael-spark-conti…
ceb47af 
…nuous-processing
@jiangmichaellll
update
9ddfcac
@jiangmichaellll
update
db2a862
@jiangmichaellll
Merge branch 'master' into jiangmichael-spark-continuous-processing
985da4b
@jiangmichaellll
partial address comments
0b094ef
@jiangmichaellll
offsets
7356b05
@jiangmichaellll
Merge branch 'jiangmichael-spark-offsets' into jiangmichael-spark-con…
0f33a8d 
…tinuous-processing
@jiangmichaellll
partial address
a38304c
@jiangmichaellll
partial address
0d33492
@jiangmichaellll
remove
78c03af
@jiangmichaellll
update
6ee4a66
@jiangmichaellll
update
13fa44f
@jiangmichaellll
address comment
7a5517d
@jiangmichaellll
address comments
dba0e12
@jiangmichaellll
update
17afb2b
@jiangmichaellll
Use blockingpullsubscriber
0167008
@jiangmichaellll
Merge branch 'jiangmichael-spark-flow-control' into jiangmichael-spar…
3bb9f83 
…k-micro-batch
@jiangmichaellll
Merge remote-tracking branch 'origin/master' into jiangmichael-spark-…
c569606 
…continuous-processing
@jiangmichaellll
address comments
b4e5541
@jiangmichaellll
address comments
705da47
@jiangmichaellll
Merge branch 'jiangmichael-spark-continuous-processing' into jiangmic…
90330ba 
…hael-spark-micro-batch
@jiangmichaellll jiangmichaellll requested a review from a team as a code owner December 14, 2020 22:08
@product-auto-label product-auto-label bot added the api: pubsublite Issues related to the googleapis/java-pubsublite API. label Dec 14, 2020
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Dec 14, 2020
yoshi-code-bot
yoshi-code-bot reviewed Dec 14, 2020
View reviewed changes
Copy link

@yoshi-code-bot yoshi-code-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some suggestions could not be made:

  • pubsublite-spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslDataSource.java
    • lines 37-38

...te-spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/HeadOffsetReader.java Outdated Show resolved Hide resolved
...blite-spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslDataSource.java Outdated Show resolved Hide resolved
...l-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchInputPartition.java Outdated Show resolved Hide resolved
...aming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchInputPartitionReader.java Outdated Show resolved Hide resolved
...spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchReader.java Outdated Show resolved Hide resolved
...-streaming/src/main/java/com/google/cloud/pubsublite/spark/RangedBlockingPullSubscriber.java Outdated Show resolved Hide resolved
yoshi-code-bot
yoshi-code-bot reviewed Dec 14, 2020
View reviewed changes
Copy link

@yoshi-code-bot yoshi-code-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some suggestions could not be made:

  • pubsublite-spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslDataSource.java
    • lines 37-38

...te-spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/HeadOffsetReader.java Outdated Show resolved Hide resolved
...blite-spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslDataSource.java Outdated Show resolved Hide resolved
...l-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchInputPartition.java Outdated Show resolved Hide resolved
...aming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchInputPartitionReader.java Outdated Show resolved Hide resolved
...spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchReader.java Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented Dec 15, 2020 •
edited

Codecov Report

Merging #426 (676a9fb) into master (36684c4) will decrease coverage by 0.58%.
The diff coverage is 48.61%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #426      +/-   ##
============================================
- Coverage     71.60%   71.01%   -0.59%     
- Complexity      876      898      +22     
============================================
  Files           164      167       +3     
  Lines          4613     4733     +120     
  Branches        234      240       +6     
============================================
+ Hits           3303     3361      +58     
- Misses         1179     1234      +55     
- Partials        131      138       +7
Impacted Files Coverage Δ Complexity Δ
...m/google/cloud/pubsublite/spark/PslDataSource.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...le/cloud/pubsublite/spark/PslContinuousReader.java 54.28% <40.00%> (-7.42%) 8.00 <0.00> (-2.00)
.../pubsublite/spark/PslMicroBatchInputPartition.java 40.00% <40.00%> (ø) 1.00 <1.00> (?)
...blite/spark/PslMicroBatchInputPartitionReader.java 62.85% <62.85%> (ø) 7.00 <7.00> (?)
...le/cloud/pubsublite/spark/PslMicroBatchReader.java 69.76% <69.76%> (ø) 12.00 <12.00> (?)
...m/google/cloud/pubsublite/spark/PslSparkUtils.java 90.69% <81.81%> (-3.06%) 9.00 <2.00> (+2.00) ⬇️
...e/cloud/pubsublite/spark/SparkPartitionOffset.java 100.00% <100.00%> (ø) 3.00 <1.00> (+1.00)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36684c4...676a9fb. Read the comment docs.

dpcollins-google
dpcollins-google requested changes Dec 15, 2020
View reviewed changes
...blite-spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslDataSource.java Outdated Show resolved Hide resolved
...l-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchInputPartition.java Outdated Show resolved Hide resolved
...aming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchInputPartitionReader.java Outdated Show resolved Hide resolved
...aming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchInputPartitionReader.java Outdated Show resolved Hide resolved
...aming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchInputPartitionReader.java Outdated Show resolved Hide resolved
...aming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchInputPartitionReader.java Outdated Show resolved Hide resolved
...spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchReader.java Outdated Show resolved Hide resolved
...spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchReader.java Outdated Show resolved Hide resolved
...spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchReader.java Outdated Show resolved Hide resolved
...spark-sql-streaming/src/main/java/com/google/cloud/pubsublite/spark/PslMicroBatchReader.java Outdated Show resolved Hide resolved
@jiangmichaellll jiangmichaellll merged commit 86aecc9 into master Dec 16, 2020
@jiangmichaellll jiangmichaellll deleted the jiangmichael-spark-micro-batch branch December 16, 2020 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yoshi-code-bot yoshi-code-bot yoshi-code-bot left review comments

@dpcollins-google dpcollins-google dpcollins-google approved these changes

Assignees

@dpcollins-google dpcollins-google

Labels
api: pubsublite Issues related to the googleapis/java-pubsublite API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@jiangmichaellll @dpcollins-google @yoshi-code-bot