chore(deps): update dependency com.google.cloud:libraries-bom to v16.1.0 #4278

renovate-bot · 2020-11-19T13:55:34Z

This PR contains the following updates:

Package	Update	Change
com.google.cloud:libraries-bom	minor	`16.0.0` -> `16.1.0`
com.google.cloud:libraries-bom	major	`15.0.0` -> `16.1.0`

Renovate configuration

📅 Schedule: At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻️ Rebasing: Never, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about these updates again.

If you want to rebase/retry this PR, check this box

This PR has been generated by WhiteSource Renovate. View repository job log here.

lesv · 2020-11-19T17:43:08Z

Java 11

@elharo FYI
@skuruppu @olavloite PTAL

------------------------------------------------------------
- testing spanner/jdbc
------------------------------------------------------------
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 2.331 s <<< FAILURE! - in com.example.spanner.jdbc.JdbcExamplesIT
[ERROR] com.example.spanner.jdbc.JdbcExamplesIT  Time elapsed: 2.3 s  <<< ERROR!
java.lang.NoClassDefFoundError: com/google/spanner/admin/instance/v1/InstanceAdminGrpc
	at cloud.spanner.jdbc@1.0.21/com.example.spanner.jdbc.JdbcExamplesIT.createTestDatabase(JdbcExamplesIT.java:82)
Caused by: java.lang.ClassNotFoundException: com.google.spanner.admin.instance.v1.InstanceAdminGrpc
	at cloud.spanner.jdbc@1.0.21/com.example.spanner.jdbc.JdbcExamplesIT.createTestDatabase(JdbcExamplesIT.java:82)

[ERROR] com.example.spanner.jdbc.JdbcExamplesIT  Time elapsed: 2.302 s  <<< ERROR!
java.lang.NullPointerException
	at cloud.spanner.jdbc@1.0.21/com.example.spanner.jdbc.JdbcExamplesIT.dropTestDatabase(JdbcExamplesIT.java:96)

[ERROR] Errors:
[ERROR] com.example.spanner.jdbc.JdbcExamplesIT.null
[ERROR]   Run 1: JdbcExamplesIT.createTestDatabase:82 » NoClassDefFound com/google/spanner/admi...
[ERROR]   Run 2: JdbcExamplesIT.dropTestDatabase:96 NullPointer
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

elharo · 2020-11-19T17:55:35Z

Interesting. I can't immediately find anything that's changed w.r.t. that class lately.

olavloite · 2020-11-19T18:05:45Z

Java 8 has the same problem:

[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 2.091 s <<< FAILURE! - in com.example.spanner.jdbc.JdbcExamplesIT
[ERROR] com.example.spanner.jdbc.JdbcExamplesIT  Time elapsed: 2.07 s  <<< ERROR!
java.lang.NoClassDefFoundError: com/google/spanner/admin/instance/v1/InstanceAdminGrpc
	at com.example.spanner.jdbc.JdbcExamplesIT.createTestDatabase(JdbcExamplesIT.java:82)
Caused by: java.lang.ClassNotFoundException: com.google.spanner.admin.instance.v1.InstanceAdminGrpc
	at com.example.spanner.jdbc.JdbcExamplesIT.createTestDatabase(JdbcExamplesIT.java:82)

[ERROR] com.example.spanner.jdbc.JdbcExamplesIT  Time elapsed: 2.071 s  <<< ERROR!
java.lang.NullPointerException
	at com.example.spanner.jdbc.JdbcExamplesIT.dropTestDatabase(JdbcExamplesIT.java:96)

[ERROR] Errors:
[ERROR] com.example.spanner.jdbc.JdbcExamplesIT.null
[ERROR]   Run 1: JdbcExamplesIT.createTestDatabase:82 » NoClassDefFound com/google/spanner/admi...
[ERROR]   Run 2: JdbcExamplesIT.dropTestDatabase:96 NullPointer

olavloite · 2020-11-19T19:49:49Z

It seems that this PR had the unintentional side effect of causing the following dependencies to no longer be considered transitive (they are no longer included from version 3.0.4 and up):

[INFO] |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:3.0.3:compile
[INFO] |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:3.0.3:compile
[INFO] |  |  \- com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:3.0.3:compile

elharo · 2020-11-19T19:56:48Z

Strange. I wouldn't have expected that to affect which dependencies are loaded, only which versions of dependencies declared elsewhere.

gcf-merge-on-green · 2020-11-19T23:44:09Z

Merge-on-green attempted to merge your PR for 6 hours, but it was not mergeable because either one of your required status checks failed, or one of your required reviews was not approved. Learn more about your required status checks here: https://help.github.com/en/github/administering-a-repository/enabling-required-status-checks. You can remove and reapply the label to re-run the bot.

suztomo · 2020-11-23T14:23:36Z

@olavloite Where did you get the missing artifacts? (The project and the directory of mvn dependency:tree and git commit?)

I compared 2 dependency trees of google-cloud-spanner-jdbc before and after googleapis/java-spanner-jdbc#258 (c9906c9). I couldn't find the 3 missing dependencies.

https://gist.github.com/suztomo/0595022143156b3b433f6cc35ea14a7c#dependency-tree-diff

olavloite · 2020-11-23T17:37:31Z

@olavloite Where did you get the missing artifacts? (The project and the directory of mvn dependency:tree and git commit?)

I compared 2 dependency trees of google-cloud-spanner-jdbc before and after googleapis/java-spanner-jdbc#258 (c9906c9). I couldn't find the 3 missing dependencies.

https://gist.github.com/suztomo/0595022143156b3b433f6cc35ea14a7c#dependency-tree-diff

@suztomo Sorry, the conclusion regarding where/when the diff was introduced was a little premature, and was mainly based on the fact that version 1.17.3 didn't contain much other changes than this, and the fact that it worked in previous versions. But: Versions 1.17.2 and 1.17.1 were never included in any boms, so they have never been tested by the samples. This meas that the previous version that was actually used by the samples was 1.17.0. I did a intersect of the commits between 1.17.0 and 1.173. to figure out where the change occurred, and it seems that it was this PR: googleapis/java-spanner-jdbc#223

One important change in 223 was that the version of the Spanner client library was bumped from 1.60.0 to 2.0.1. Strangely enough it seems that:

In 1.60.0 grpc-google-cloud-spanner-admin-database-v1 was a TEST dependency
In 2.0.1 grpc-google-cloud-spanner-admin-database-v1 was a COMPILE depdency

Which would have me expect that the problem should have been present in 1.60.0 and not in 2.0.1, but in reality it seems to be the other way around...

elharo · 2020-11-23T18:00:58Z

Your investigation suggests to me that the problem is likely in the samples, not google-cloud-spanner. Something in the samples is relying on a transitive dependency that was removed between versions.

The other possibility is that google-cloud-spanner is missing a runtime dependency. I don't see any way it could be missing a compile dependency.

elharo · 2020-11-23T18:05:03Z

This is the line that's failing

      Iterator<Instance> iterator =
          spanner.getInstanceAdminClient().listInstances().iterateAll().iterator();

getInstanceAdminClient returns a com.google.cloud.spanner.InstanceAdminClient.

olavloite · 2020-11-23T18:22:48Z

Your investigation suggests to me that the problem is likely in the samples, not google-cloud-spanner. Something in the samples is relying on a transitive dependency that was removed between versions.

The other possibility is that google-cloud-spanner is missing a runtime dependency. I don't see any way it could be missing a compile dependency.

I think the problem is in google-cloud-spanner-jdbc. In the current situation

google-cloud-spanner defines grpc-google-cloud-spanner-admin-instance-v1 and grpc-google-cloud-spanner-admin-database-v1 as COMPILE time dependencies (i.e. also transitive). These also need to be transitive, as anyone using the Spanner client library will (most likely) also use features from these libraries.
google-cloud-spanner-jdbc for some reason re-declares the above dependencies as TEST dependencies. That did not affect the transitive nature of these dependencies up until version 1.17.0 of google-cloud-spanner-jdbc.
From version 1.17.3 the re-declaration of those dependencies as TEST dependencies seems to stop grpc-google-cloud-spanner-admin-instance-v1 and grpc-google-cloud-spanner-admin-database-v1 from being transitive, while they are still found on the compile classpath of google-cloud-spanner-jdbc.

Those dependencies should really be transitive and it should not be necessary for anyone using google-cloud-spanner-jdbc to have to explicitly include them. Without grpc-google-cloud-spanner-admin-database-v1 it would for example be impossible to execute DDL statements with the JDBC driver.

https://github.com/googleapis/java-spanner-jdbc/pull/278/files now removes the re-declaration of the grpc-... dependencies in the JDBC driver, and that solves the problem locally.

elharo · 2020-11-23T19:03:06Z

"Those dependencies should really be transitive and it should not be necessary for anyone using google-cloud-spanner-jdbc to have to explicitly include them. Without grpc-google-cloud-spanner-admin-database-v1 it would for example be impossible to execute DDL statements with the JDBC driver."

If that's the case, then they probably should never have been put in separate artifacts in the first place. There might also be an ill-advised interface/implementation split here as well.

However assuming that ship has sailed and can't be fixed now, grpc-google-cloud-spanner-admin-database-v1 still shouldn't be a compile scoped dependency of google-cloud-spanner-jdbc (unless it's needed to compile google-cloud-spanner-jdbc which doesn't seem to be the case.) At most it should be a runtime scoped dependency, and even that is iffy.

olavloite · 2020-11-23T21:26:48Z

"Those dependencies should really be transitive and it should not be necessary for anyone using google-cloud-spanner-jdbc to have to explicitly include them. Without grpc-google-cloud-spanner-admin-database-v1 it would for example be impossible to execute DDL statements with the JDBC driver."

If that's the case, then they probably should never have been put in separate artifacts in the first place. There might also be an ill-advised interface/implementation split here as well.

Sorry, I'm not sure I understand what you mean in this case:

grpc-google-cloud-spanner-admin-database-v1 and grpc-google-cloud-spanner-admin-instance-v1 only contain generated code from the proto definitions.
google-cloud-spanner-jdbc and google-cloud-spanner could in theory have been one artifact, but that is most probably a ship that has sailed.

However assuming that ship has sailed and can't be fixed now, grpc-google-cloud-spanner-admin-database-v1 still shouldn't be a compile scoped dependency of google-cloud-spanner-jdbc (unless it's needed to compile google-cloud-spanner-jdbc which doesn't seem to be the case.) At most it should be a runtime scoped dependency, and even that is iffy.

Sorry, I'm confused, so please bare with me if I'm missing your whole point. As far as I see it, the dependency tree should be as simple as this:

grpc-google-cloud-spanner-admin-database-v1 is part of the generated Spanner gapic client.
google-cloud-spanner (The Spanner client library) has a compile time dependency on grpc-google-cloud-spanner-admin-database-v1. That dependency is needed to execute any 'admin' operations on Spanner, e.g. DDL statements.
google-cloud-spanner-jdbc (The Spanner JDBC Driver) has a compile time dependency on google-cloud-spanner. That means that grpc-google-cloud-spanner-admin-database-v1 is also added as a compile time dependency to google-cloud-spanner-jdbc, as it is a transitive dependency of google-cloud-spanner.

(The problem that caused this build error was that the transitive nature of the dependency was somehow broken between step 2 and 3)

I don't see any reason why the above should be treated any differently from:

com.google.cloud:gax is a compile time dependency of google-cloud-spanner.
google-cloud-spanner-jdbc has a compile time dependency on google-cloud-spanner.
com.google.cloud:gax is also a (transitive) compile time dependency of google-cloud-spanner-jdbc, and is not something that any user of the JDBC driver needs to add as a dependency to their applications.

suztomo · 2020-11-23T22:40:55Z

google-cloud-spanner (The Spanner client library) has a compile time dependency on grpc-google-cloud-spanner-admin-database-v1.

I think the exclusion below is what we're looking for. The published pom.xml of google-cloud-spanner-jdbc (https://search.maven.org/artifact/com.google.cloud/google-cloud-spanner-jdbc/1.17.3/jar) excludes grpc-google-cloud-spanner-admin-database-v1:

   <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-spanner</artifactId>
      <version>2.0.2</version>
      <scope>compile</scope>
      <exclusions>
...
        <exclusion>
          <artifactId>grpc-google-cloud-spanner-admin-database-v1</artifactId>
          <groupId>com.google.api.grpc</groupId>
        </exclusion>

(The pom.xml at v1.17.3 does not declare these exclusions)

Because of this exclusion, the dependencies of google-cloud-spanner do not appear in the dependency tree of google-cloud-spanner-jdbc. Therefore classes in google-cloud-spanner would fail for the missing classes.

@stephaniewang526 Would you validate my finding above?

elharo · 2020-11-23T22:47:53Z

"google-cloud-spanner (The Spanner client library) has a compile time dependency on grpc-google-cloud-spanner-admin-database-v1. That dependency is needed to execute any 'admin' operations on Spanner, e.g. DDL statements."

Is the grpc-google-cloud-spanner-admin-database-v1 dependency needed to compile google-cloud-spanner? If it isn't, it should not be a compile dependency of google-cloud-spanner. If it's needed to run google-cloud-spanner then make it a runtime dependency, not a compile scope dependency.

olavloite · 2020-11-24T05:54:37Z

"google-cloud-spanner (The Spanner client library) has a compile time dependency on grpc-google-cloud-spanner-admin-database-v1. That dependency is needed to execute any 'admin' operations on Spanner, e.g. DDL statements."

Is the grpc-google-cloud-spanner-admin-database-v1 dependency needed to compile google-cloud-spanner? If it isn't, it should not be a compile dependency of google-cloud-spanner. If it's needed to run google-cloud-spanner then make it a runtime dependency, not a compile scope dependency.

Yes. Changing it to a runtime dependency would cause several compile errors, for example in https://github.com/googleapis/java-spanner/blob/master/google-cloud-spanner/src/main/java/com/google/cloud/spanner/DatabaseAdminClientImpl.java

elharo · 2020-11-24T12:03:05Z

Thanks. That clears some things up. In that case, @suztomo is likely correct. There's an exclusion in play that shouldn't be there.

elharo · 2020-11-24T12:10:25Z

The question is, why are those exclusions there? They aren't in the source pom.xml in Github. Weird. This is what we need to understand and ifx.

suztomo · 2020-11-24T17:04:48Z

Stephanie found this test-scope declaration (before flattened).

https://github.com/googleapis/java-spanner-jdbc/blob/9b1b9067241736c7e2f6d75d3f34824bc4ea1aca/pom.xml#L173

olavloite · 2020-11-24T17:11:14Z

Stephanie found this test-scope declaration (before flattened).

https://github.com/googleapis/java-spanner-jdbc/blob/9b1b9067241736c7e2f6d75d3f34824bc4ea1aca/pom.xml#L173

Yes, my assumption is also that that is what is causing the problem. Although I don't quite understand why it did not cause the problem previously, but does now. googleapis/java-spanner-jdbc#278 removes the test dependencies, and I had already verified locally that it solves this problem.

lesv · 2020-12-07T18:39:55Z

I'm going to close this as it looks like we'll need an updated BOM before it's fixed.

forking-renovate · 2020-12-07T18:43:36Z

Renovate Ignore Notification

As this PR has been closed unmerged, Renovate will now ignore this update (16.1.0). You will still receive a PR once a newer version is released, so if you wish to permanently ignore this dependency, please add it to the ignoreDeps array of your renovate config.

If this PR was closed by mistake or you changed your mind, you can simply rename this PR and you will soon get a fresh replacement PR opened.

renovate-bot requested a review from a team November 19, 2020 13:55

forking-renovate bot added the automerge Merge the pull request once unit tests and other checks pass. label Nov 19, 2020

trusted-contributions-gcf bot added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 19, 2020

google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Nov 19, 2020

product-auto-label bot added the samples Issues that are directly related to samples. label Nov 19, 2020

blunderbuss-gcf bot assigned lesv Nov 19, 2020

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 19, 2020

lesv approved these changes Nov 19, 2020

View reviewed changes

lesv added api: spanner Issues related to the Spanner API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Nov 19, 2020

olavloite self-assigned this Nov 19, 2020

olavloite mentioned this pull request Nov 19, 2020

deps: do not re-declare grpc dependencies as test dependencies googleapis/java-spanner-jdbc#278

Merged

gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Nov 19, 2020

suztomo mentioned this pull request Nov 23, 2020

Linkage Monitor failed to catch missing transitive dependencies of google-cloud-spanner-jdbc GoogleCloudPlatform/cloud-opensource-java#1831

Closed

chore(deps): update dependency com.google.cloud:libraries-bom to v16.1.0

e01bc6a

renovate-bot force-pushed the renovate/com.google.cloud-libraries-bom-16.x branch from db57a03 to e01bc6a Compare November 25, 2020 23:14

trusted-contributions-gcf bot added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 25, 2020

kokoro-team removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 25, 2020

suztomo mentioned this pull request Nov 30, 2020

Not to exclude a test dependency if the dependency is used in another dependency mojohaus/flatten-maven-plugin#185

Closed

lesv closed this Dec 7, 2020

renovate-bot deleted the renovate/com.google.cloud-libraries-bom-16.x branch December 7, 2020 18:43

lesv mentioned this pull request Dec 7, 2020

iot/api-client/manager & com.sun.xml.bind:jaxb-core v3 issues #4354

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore(deps): update dependency com.google.cloud:libraries-bom to v16.1.0 #4278

chore(deps): update dependency com.google.cloud:libraries-bom to v16.1.0 #4278

renovate-bot commented Nov 19, 2020

lesv commented Nov 19, 2020

elharo commented Nov 19, 2020

olavloite commented Nov 19, 2020

olavloite commented Nov 19, 2020

elharo commented Nov 19, 2020

gcf-merge-on-green bot commented Nov 19, 2020

suztomo commented Nov 23, 2020 •

edited

olavloite commented Nov 23, 2020

elharo commented Nov 23, 2020

elharo commented Nov 23, 2020

olavloite commented Nov 23, 2020

elharo commented Nov 23, 2020 •

edited

olavloite commented Nov 23, 2020

suztomo commented Nov 23, 2020 •

edited

elharo commented Nov 23, 2020

olavloite commented Nov 24, 2020

elharo commented Nov 24, 2020

elharo commented Nov 24, 2020

suztomo commented Nov 24, 2020

olavloite commented Nov 24, 2020

lesv commented Dec 7, 2020

forking-renovate bot commented Dec 7, 2020

chore(deps): update dependency com.google.cloud:libraries-bom to v16.1.0 #4278

chore(deps): update dependency com.google.cloud:libraries-bom to v16.1.0 #4278

Conversation

renovate-bot commented Nov 19, 2020

Renovate configuration

lesv commented Nov 19, 2020

Java 11

elharo commented Nov 19, 2020

olavloite commented Nov 19, 2020

olavloite commented Nov 19, 2020

elharo commented Nov 19, 2020

gcf-merge-on-green bot commented Nov 19, 2020

suztomo commented Nov 23, 2020 • edited

olavloite commented Nov 23, 2020

elharo commented Nov 23, 2020

elharo commented Nov 23, 2020

olavloite commented Nov 23, 2020

elharo commented Nov 23, 2020 • edited

olavloite commented Nov 23, 2020

suztomo commented Nov 23, 2020 • edited

elharo commented Nov 23, 2020

olavloite commented Nov 24, 2020

elharo commented Nov 24, 2020

elharo commented Nov 24, 2020

suztomo commented Nov 24, 2020

olavloite commented Nov 24, 2020

lesv commented Dec 7, 2020

forking-renovate bot commented Dec 7, 2020

Renovate Ignore Notification

suztomo commented Nov 23, 2020 •

edited

elharo commented Nov 23, 2020 •

edited

suztomo commented Nov 23, 2020 •

edited