Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: create manual client for Write API #112

Merged
merged 36 commits into from Apr 3, 2020
Merged

feat: create manual client for Write API #112

merged 36 commits into from Apr 3, 2020

Conversation

yirutang
Copy link
Contributor

Fixes #111 ☕️

A StreamWriter that provides built-in capabilities to help making bi-di streaming client easy to use:

  • handle batching of messages;
  • controlling memory utilization (through flow control);
  • automatic connection re-establishment;
  • request cleanup (only keeps write schema on first request in the stream).

Example:
WriteStream writeStream =
client.createWriteStream(
CreateWriteStreamRequest.newBuilder()
.setParent(tableId2)
.setWriteStream(WriteStream.newBuilder().setType(WriteStream.Type.PENDING).build())
.build());
StreamWriter streamWriter = StreamWriter.newBuilder(writeStream.getName()).build();
ApiFuture response =
streamWriter.append(
createAppendRequestComplicateType(writeStream.getName(), new String[] {"aaa"})
.setOffset(Int64Value.of(0L))
.build());
assertEquals(0, response.get().getOffset());
streamWriter.shutdown();
}

First version, test to be developed.
…brary helps to convert a ProtoDescriptorProto out of ProtoDescriptor into a self contained ProtoDescriptorProto, that can be passed into API and reconstructed on the server side.

	modified:   google-cloud-bigquerystorage/pom.xml
	new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverter.java
	new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverterTest.java
	new file:   google-cloud-bigquerystorage/src/test/proto/test.proto
	modified:   pom.xml
First version, test to be developed.
.
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWriteImpl.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeScheduledExecutorService.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
@googlebot googlebot added the cla: yes This human has signed the Contributor License Agreement. label Mar 19, 2020
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
@yirutang yirutang changed the title Create manual client for Write API feat: create manual client for Write API Mar 19, 2020
@stephaniewang526 stephaniewang526 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 20, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Mar 20, 2020
@codecov
Copy link

codecov bot commented Mar 20, 2020

Codecov Report

Merging #112 into master will increase coverage by 1.52%.
The diff coverage is 81.05%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #112      +/-   ##
============================================
+ Coverage     72.36%   73.88%   +1.52%     
- Complexity      281      319      +38     
============================================
  Files            39       42       +3     
  Lines          1809     2229     +420     
  Branches          4       70      +66     
============================================
+ Hits           1309     1647     +338     
- Misses          496      552      +56     
- Partials          4       30      +26
Impacted Files Coverage Δ Complexity Δ
...google/cloud/bigquery/storage/v1alpha2/Waiter.java 72.54% <72.54%> (ø) 14 <14> (?)
.../cloud/bigquery/storage/v1alpha2/StreamWriter.java 81.17% <81.17%> (ø) 28 <28> (?)
...igquery/storage/v1alpha2/ProtoSchemaConverter.java 93.54% <93.54%> (ø) 6 <6> (?)
...igquery/storage/v1beta1/BigQueryStorageClient.java 68.75% <0%> (-1.84%) 16% <0%> (-3%)
...ery/storage/v1beta1/BaseBigQueryStorageClient.java 68.75% <0%> (-1.84%) 16% <0%> (-3%)
...d/bigquery/storage/v1beta2/BigQueryReadClient.java 51.51% <0%> (-1.43%) 8% <0%> (-1%)
.../cloud/bigquery/storage/v1/BigQueryReadClient.java 51.51% <0%> (-1.43%) 8% <0%> (-1%)
...ud/bigquery/storage/v1/BaseBigQueryReadClient.java 54.54% <0%> (-1.34%) 9% <0%> (-1%)
...gquery/storage/v1beta2/BaseBigQueryReadClient.java 54.54% <0%> (-1.34%) 9% <0%> (-1%)
...ery/storage/v1beta2/stub/GrpcBigQueryReadStub.java 89.74% <0%> (-0.62%) 9% <0%> (-1%)
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c4814dc...f942f48. Read the comment docs.

@stephaniewang526
Copy link
Contributor

please take a look at the various build failures.

.
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
…work

	modified:   google-cloud-bigquerystorage/pom.xml
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
	modified:   google-cloud-bigquerystorage/src/test/proto/test.proto

	modified:   google-cloud-bigquerystorage/pom.xml
	modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
	modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java
	modified:   google-cloud-bigquerystorage/src/test/proto/test.proto
@stephaniewang526
Copy link
Contributor

As a general note, all public methods need to be documented clearly in accompanying Java doc to help developers understand what the inputs, outputs, exceptions, etc are. I see a number of public methods without Java doc. Please make sure to put them in.

yirutang and others added 5 commits April 1, 2020 10:53
…gquery/storage/v1alpha2/Waiter.java

Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>
…gquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>
…gquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>
Copy link
Contributor Author

@yirutang yirutang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PTAL

@stephaniewang526 stephaniewang526 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 2, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 2, 2020
@stephaniewang526 stephaniewang526 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 2, 2020
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Apr 2, 2020
@stephaniewang526 stephaniewang526 merged commit 98851e9 into googleapis:master Apr 3, 2020
yoshi-automation added a commit that referenced this pull request Apr 3, 2020
98851e9
commit 98851e9
Author: Yiru Tang <yiru@google.com>
Date:   Thu Apr 2 17:45:37 2020 -0700

    feat: create manual client for Write API (#112)

    * First writeapi manual client

    First version, test to be developed.

    * Manual client with e2e

    * StreamWriter library

    * ProtoSchema convert library WriteApi client library addition. This library helps to convert a ProtoDescriptorProto out of ProtoDescriptor into a self contained ProtoDescriptorProto, that can be passed into API and reconstructed on the server side.

            modified:   google-cloud-bigquerystorage/pom.xml
            new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverter.java
            new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverterTest.java
            new file:   google-cloud-bigquerystorage/src/test/proto/test.proto
            modified:   pom.xml

    * First writeapi manual client

    First version, test to be developed.

    * .

    * incremental development

    * .
            modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWriteImpl.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeScheduledExecutorService.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    * .

    * .

    * .

    * feat: create manual java client for writeapi

            modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java

    * feat: add integration test final verification.

    * .

    * .

            modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    * .

    * .

    * .

    * .

    * avoid duplicate count++ and make it easier to read

    * Update google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/Waiter.java

    Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

    * Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

    * Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

    * .

    * .

    * .

    * .

    * .

    * dep: add undeclared dep in parent pom

    * dep: remove duplicate and add undeclared

    remove dupe gax; add undeclared commons-lang3

    Co-authored-by: Stephanie Wang <stephaniewang526@users.noreply.github.com>
stephaniewang526 pushed a commit that referenced this pull request Apr 3, 2020
…nerator. (#134)

* feat: create manual client for Write API (#112)

98851e9
commit 98851e9
Author: Yiru Tang <yiru@google.com>
Date:   Thu Apr 2 17:45:37 2020 -0700

    feat: create manual client for Write API (#112)

    * First writeapi manual client

    First version, test to be developed.

    * Manual client with e2e

    * StreamWriter library

    * ProtoSchema convert library WriteApi client library addition. This library helps to convert a ProtoDescriptorProto out of ProtoDescriptor into a self contained ProtoDescriptorProto, that can be passed into API and reconstructed on the server side.

            modified:   google-cloud-bigquerystorage/pom.xml
            new file:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverter.java
            new file:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/ProtoSchemaConverterTest.java
            new file:   google-cloud-bigquerystorage/src/test/proto/test.proto
            modified:   pom.xml

    * First writeapi manual client

    First version, test to be developed.

    * .

    * incremental development

    * .
            modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeBigQueryWriteImpl.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/FakeScheduledExecutorService.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriterTest.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    * .

    * .

    * .

    * feat: create manual java client for writeapi

            modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java

    * feat: add integration test final verification.

    * .

    * .

            modified:   google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/StreamWriter.java
            modified:   google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    * .

    * .

    * .

    * .

    * avoid duplicate count++ and make it easier to read

    * Update google-cloud-bigquerystorage/src/main/java/com/google/cloud/bigquery/storage/v1alpha2/Waiter.java

    Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

    * Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

    * Update google-cloud-bigquerystorage/src/test/java/com/google/cloud/bigquery/storage/v1alpha2/it/ITBigQueryWriteManualClientTest.java

    Co-Authored-By: Stephanie Wang <stephaniewang526@users.noreply.github.com>

    * .

    * .

    * .

    * .

    * .

    * dep: add undeclared dep in parent pom

    * dep: remove duplicate and add undeclared

    remove dupe gax; add undeclared commons-lang3

    Co-authored-by: Stephanie Wang <stephaniewang526@users.noreply.github.com>

* deps: update dependency io.grpc:grpc-bom to v1.28.1 (#132)

711c5c7
commit 711c5c7
Author: WhiteSource Renovate <bot@renovateapp.com>
Date:   Fri Apr 3 03:08:41 2020 +0200

    deps: update dependency io.grpc:grpc-bom to v1.28.1 (#132)

* chore: release 0.127.0 (#122)

45fc919
commit 45fc919
Author: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Date:   Thu Apr 2 21:49:46 2020 -0400

    chore: release 0.127.0 (#122)

    * updated CHANGELOG.md [ci skip]

    * updated README.md [ci skip]

    * updated versions.txt [ci skip]

    * updated google-cloud-bigquerystorage-bom/pom.xml [ci skip]

    * updated grpc-google-cloud-bigquerystorage-v1/pom.xml [ci skip]

    * updated grpc-google-cloud-bigquerystorage-v1alpha2/pom.xml [ci skip]

    * updated grpc-google-cloud-bigquerystorage-v1beta1/pom.xml [ci skip]

    * updated grpc-google-cloud-bigquerystorage-v1beta2/pom.xml [ci skip]

    * updated proto-google-cloud-bigquerystorage-v1/pom.xml [ci skip]

    * updated proto-google-cloud-bigquerystorage-v1alpha2/pom.xml [ci skip]

    * updated proto-google-cloud-bigquerystorage-v1beta1/pom.xml [ci skip]

    * updated proto-google-cloud-bigquerystorage-v1beta2/pom.xml [ci skip]

    * updated samples/install-without-bom/pom.xml [ci skip]

    * updated samples/pom.xml [ci skip]

    * updated samples/snapshot/pom.xml [ci skip]

    * updated samples/snippets/pom.xml [ci skip]

    * updated google-cloud-bigquerystorage/pom.xml [ci skip]

    * updated pom.xml [ci skip]

    Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
@yirutang yirutang deleted the work branch June 5, 2020 17:36
shubhwip pushed a commit to shubhwip/java-bigquerystorage that referenced this pull request Oct 7, 2023
….0 (googleapis#112)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [com.google.cloud:libraries-bom](https://togithub.com/GoogleCloudPlatform/cloud-opensource-java) | minor | `5.3.0` -> `5.4.0` |

---

### Renovate configuration

:date: **Schedule**: At any time (no schedule defined).

:vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

:recycle: **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

:no_bell: **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#googleapis/java-storage-nio).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create a manual java client to help manage bi direction streaming client for Write API
5 participants