feat: Added schema compatibility check functionality (SchemaCompact.java) #339

allenc3 · 2020-06-10T21:42:53Z

Finished schema compatibility check class, and added a proto file that is used for testing.

Modified WriterCache.java, DirectWriterTest.java, and WriterCacheTest.java because checking for schema compatibility requires three parameters instead of two, and so the "check" method requires three parameters.

codecov · 2020-06-10T21:44:52Z

Codecov Report

Merging #339 into master will increase coverage by 0.84%.
The diff coverage is 93.70%.

@@             Coverage Diff              @@
##             master     #339      +/-   ##
============================================
+ Coverage     75.63%   76.47%   +0.84%     
- Complexity      465      555      +90     
============================================
  Files            54       54              
  Lines          3123     3261     +138     
  Branches        175      196      +21     
============================================
+ Hits           2362     2494     +132     
- Misses          655      658       +3     
- Partials        106      109       +3

Impacted Files	Coverage Δ	Complexity Δ
...cloud/bigquery/storage/v1alpha2/SchemaCompact.java	`91.19% <93.70%> (+15.00%)`	`96.00 <91.00> (+89.00)`
.../cloud/bigquery/storage/v1alpha2/StreamWriter.java	`81.07% <0.00%> (+0.70%)`	`34.00% <0.00%> (+1.00%)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d42ecd...a6e3867. Read the comment docs.

...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompact.java

google-cloud-bigquerystorage/clirr-ignored-differences.xml

...querystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompactTest.java

...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompact.java

googlebot · 2020-06-15T18:10:34Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to determine that you authored the commits in this PR. Maybe you used a different email address in the git commits than was used to sign the CLA? If someone else authored these commits, then please add them to this pull request and have them confirm that they're okay with them being contributed to Google. If there are co-authors, make sure they're formatted properly.

In order to pass this check, please resolve this problem and then comment@googlebot I fixed it... If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

googlebot · 2020-06-15T18:41:28Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

yirutang

Almost there, just one more test case requested.

...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompact.java

...querystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompactTest.java

...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompact.java

stephaniewang526 · 2020-06-18T02:15:16Z

google-cloud-bigquerystorage/clirr-ignored-differences.xml

nothing has changed - please revert this file.

stephaniewang526 · 2020-06-18T02:15:48Z

...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompact.java


  private SchemaCompact(BigQuery bigquery) {
+    // TODO: Add functionality that allows SchemaCompact to build schemas.


is there a reason for us to keep this TODO here?

Yup, Yiru asked me to leave that there as that should be a functionality we should add in, but not related to the current PR.

…ma types are supported by BQ.

…s on scalar types and oneof types

…pendency in pom.

…g method

…otations.

…th tests to account for BQRecord

…to schemas and BQ schemas.

…hod signature

…PR suggestions.

…Info as SchemaCheck does not allow BQ fields with no field mode.

🤖 I have created a release \*beep\* \*boop\* --- ## [0.134.0](https://www.github.com/googleapis/java-bigquerystorage/compare/v0.133.4...v0.134.0) (2020-06-18) ### Features * Added schema compatibility check functionality (SchemaCompact.java) ([#339](https://www.github.com/googleapis/java-bigquerystorage/issues/339)) ([bc2d8cc](https://www.github.com/googleapis/java-bigquerystorage/commit/bc2d8cc82adeeddb21aeb9845e0883d369101513)) ### Dependencies * update dependency com.google.cloud:google-cloud-shared-dependencies to v0.8.1 ([#346](https://www.github.com/googleapis/java-bigquerystorage/issues/346)) ([9db4c47](https://www.github.com/googleapis/java-bigquerystorage/commit/9db4c475118b03b323b4dc502ec0634692e9dea3)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please).

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jun 10, 2020

allenc3 force-pushed the SchemaCompactNoAnnotatation branch from 7a36964 to 9bacc48 Compare June 11, 2020 15:57

yirutang reviewed Jun 11, 2020

View reviewed changes

allenc3 force-pushed the SchemaCompactNoAnnotatation branch from dcb2727 to e8b3e68 Compare June 12, 2020 15:48

yirutang reviewed Jun 12, 2020

View reviewed changes

...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompact.java Outdated Show resolved Hide resolved

HowAreYouMyFriend suggested changes Jun 12, 2020

View reviewed changes

googlebot added cla: no This human has *not* signed the Contributor License Agreement. and removed cla: yes This human has signed the Contributor License Agreement. labels Jun 15, 2020

allenc3 force-pushed the SchemaCompactNoAnnotatation branch from cd4d7ac to 3671801 Compare June 15, 2020 18:40

googlebot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Jun 15, 2020

yirutang reviewed Jun 15, 2020

View reviewed changes

allenc3 force-pushed the SchemaCompactNoAnnotatation branch 2 times, most recently from 3a009dc to 3a03d46 Compare June 16, 2020 17:04

yirutang approved these changes Jun 16, 2020

View reviewed changes

...querystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompactTest.java Show resolved Hide resolved

allenc3 force-pushed the SchemaCompactNoAnnotatation branch 2 times, most recently from 892a3eb to c1ceb60 Compare June 17, 2020 19:25

HowAreYouMyFriend approved these changes Jun 17, 2020

View reviewed changes

...-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/SchemaCompact.java Outdated Show resolved Hide resolved

stephaniewang526 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 18, 2020

stephaniewang526 suggested changes Jun 18, 2020

View reviewed changes

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 18, 2020

allenc3 and others added 6 commits June 18, 2020 15:34

Protobuf and annotations working; first test that chekcs if user sche…

059dbf8

…ma types are supported by BQ.

Halfway through basic type checking of nonsupported types; wrote test…

38d779e

…s on scalar types and oneof types

Almost finished detecting cycles in protos.

7888b32

Finished checking for nonsupported types, added zetasql package as de…

c6b939c

…pendency in pom.

Reformatted to pass checkstyle.

cdff32f

Added all BQ compatibility checks and started working on main checkin…

087278f

…g method

allenc3 added 23 commits June 18, 2020 15:34

Added field mode checks

0ce7d3a

Refactored field option checking code.

816d3d0

All current tests work.

3219d9e

Finished BQInteger testing; need to add testing for all types and ann…

bc390e3

…otations.

Removing zetasql related code

eabfd69

Removed all zetasql related code

865c582

Finished basic type checking and added tests, need to add more in-dep…

0590580

…th tests to account for BQRecord

feat: Added SchemaCompact class that checks compatibility between pro…

1896159

…to schemas and BQ schemas.

Changed some files to make compilation/testing work

106fe4c

Added to clirr-ignored-differences to ignore differences in check met…

ff556a2

…hod signature

Changed clirr ignored method to be the original method signature

5f7287b

Fixed code for pull request

c94505c

Fixed linting errors

ef3dfd7

Fixed all PR suggestions

301f070

Merged isSupported and isCompatible, fixed minor issues according to …

822b3ef

…PR suggestions.

Removed duplicate commits

f4c4da5

Fix linting

c865802

Added a testcase for field name casing.

8dcd0c0

Added Integration tests, also added a field mode for footype in table…

2e9a30c

…Info as SchemaCheck does not allow BQ fields with no field mode.

Fixed linting

74fb3b5

Fixed testing errors

013a9db

Fixing test pt 2

5fbd899

Reverted clirr-ignored-differences.ml

a6e3867

allenc3 force-pushed the SchemaCompactNoAnnotatation branch from c15a329 to a6e3867 Compare June 18, 2020 15:35

stephaniewang526 approved these changes Jun 18, 2020

View reviewed changes

allenc3 marked this pull request as ready for review June 18, 2020 15:44

stephaniewang526 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 18, 2020

yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Jun 18, 2020

stephaniewang526 merged commit bc2d8cc into googleapis:master Jun 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Added schema compatibility check functionality (SchemaCompact.java) #339

feat: Added schema compatibility check functionality (SchemaCompact.java) #339

allenc3 commented Jun 10, 2020

codecov bot commented Jun 10, 2020 •

edited

googlebot commented Jun 15, 2020

googlebot commented Jun 15, 2020

yirutang left a comment

stephaniewang526 Jun 18, 2020

stephaniewang526 Jun 18, 2020

allenc3 Jun 18, 2020

stephaniewang526 Jun 18, 2020


		private SchemaCompact(BigQuery bigquery) {
		// TODO: Add functionality that allows SchemaCompact to build schemas.

feat: Added schema compatibility check functionality (SchemaCompact.java) #339

feat: Added schema compatibility check functionality (SchemaCompact.java) #339

Conversation

allenc3 commented Jun 10, 2020

codecov bot commented Jun 10, 2020 • edited

Codecov Report

googlebot commented Jun 15, 2020

googlebot commented Jun 15, 2020

yirutang left a comment

Choose a reason for hiding this comment

stephaniewang526 Jun 18, 2020

Choose a reason for hiding this comment

stephaniewang526 Jun 18, 2020

Choose a reason for hiding this comment

allenc3 Jun 18, 2020

Choose a reason for hiding this comment

stephaniewang526 Jun 18, 2020

Choose a reason for hiding this comment

codecov bot commented Jun 10, 2020 •

edited