Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Update spark connector pom.xml #445

Merged
merged 3 commits into from Jan 11, 2021

Conversation

jiangmichaellll
Copy link
Contributor

@jiangmichaellll jiangmichaellll commented Jan 8, 2021

  1. use auto service instead of auto service annotation in the processor. https://screenshot.googleplex.com/AZCi8QjvdQ9AdC2
  2. Add relocations to resolve grpc-netty-shaded issues.

@jiangmichaellll jiangmichaellll requested review from a team as code owners January 8, 2021 21:36
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Jan 8, 2021
@product-auto-label product-auto-label bot added the api: pubsublite Issues related to the googleapis/java-pubsublite API. label Jan 8, 2021
@codecov
Copy link

codecov bot commented Jan 8, 2021

Codecov Report

Merging #445 (a4ed1a6) into master (5190989) will decrease coverage by 0.23%.
The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #445      +/-   ##
============================================
- Coverage     71.32%   71.08%   -0.24%     
  Complexity      913      913              
============================================
  Files           167      167              
  Lines          4812     4831      +19     
  Branches        244      244              
============================================
+ Hits           3432     3434       +2     
- Misses         1241     1257      +16     
- Partials        139      140       +1     
Impacted Files Coverage Δ Complexity Δ
...le/cloud/pubsublite/spark/PslContinuousReader.java 54.05% <0.00%> (-0.24%) 8.00% <0.00%> (ø%)
...m/google/cloud/pubsublite/spark/PslDataSource.java 0.00% <0.00%> (ø) 0.00% <0.00%> (ø%)
...loud/pubsublite/internal/wire/RoutingMetadata.java 0.00% <0.00%> (ø) 0.00% <0.00%> (ø%)
.../pubsublite/spark/PslContinuousInputPartition.java 0.00% <0.00%> (ø) 0.00% <0.00%> (ø%)
...ogle/cloud/pubsublite/internal/ServiceClients.java
...cloud/pubsublite/internal/wire/ServiceClients.java 0.00% <0.00%> (ø) 0.00% <0.00%> (?%)
...oud/pubsublite/internal/wire/PublisherBuilder.java 68.75% <0.00%> (+6.48%) 3.00% <0.00%> (ø%)
...ud/pubsublite/internal/wire/SubscriberBuilder.java 47.82% <0.00%> (+7.08%) 2.00% <0.00%> (ø%)
...le/cloud/pubsublite/spark/PslMicroBatchReader.java 77.50% <0.00%> (+7.73%) 12.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5190989...c780c14. Read the comment docs.

@jiangmichaellll
Copy link
Contributor Author

jiangmichaellll commented Jan 11, 2021

I am going to try another suggestion from dataproc team. This PR makes it working in dataproc but restrains the fat jar with specific versions of tcnative that might not work with other images.

EDIT: the grpc netty shaded relocation works like a charm!

pubsublite-spark-sql-streaming/pom.xml Outdated Show resolved Hide resolved
pubsublite-spark-sql-streaming/pom.xml Outdated Show resolved Hide resolved
@jiangmichaellll jiangmichaellll merged commit 624b123 into master Jan 11, 2021
@jiangmichaellll jiangmichaellll deleted the jiangmichael-spark-correct-versions branch January 11, 2021 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: pubsublite Issues related to the googleapis/java-pubsublite API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants