Skip to content

Commit

Permalink
feature: Added JsonStreamWriter, which is built on top of a StreamWri…
Browse files Browse the repository at this point in the history
…ter and writes Json data by first converting to protobuf messages then calling StreamWriter's append (#437)

* Added JsonStreamWriter

* Finished basics of JsonStreamWriter

* Working basic JsonStreamWriter

* Attempt to implement schema update

* Progress on update schema

* Finished designing of JsonStreamWriter, waiting for JsonToProtoMessage to be approved.

* Update append method to convert json data to proto messages.

* Temp commit, somehow missing protos

* Copied over protos

* Added more tests and finalize JsonStreamWriter design

* ls

* Finished tests

* Add import for Optional

* Removed Optional

* Add multiple append test case

* Added test case for schema update exception

* Adding offset exception test cases

* Fixing handling exception cases

* Add more test case and make sure JsonStreamWriter and StreamWriter are correct

* Added IT test

* Changed schema update callback to be a runnable and adding ITTests

* Changed variables to be private and added getter methods for runnable, changing IT tests now

* update to IT test

* Finished JsonStreamWriter, need to fix e2e for schema update.

* fix minor detail in JsonStreamWriter

* More it Tests

* Fixed according to PR suggestions

* Trigger rerun

* Trigger rerun

* Fixed according to suggestions

* modified IT test

* temp commit

* Added e2e schema update test, multi thread append JsonStreamWriterTest, and fixed refreshAppend in StreamWriter

* Remove commentted out code

* Experimenting on writer schema

* Fixed according to PR suggestions; dealth with syncrhonization issues by letting StreamWriter handle sending first writer schema, added an additional lock between append and refreshAppend for StreamWriter

* Fix null ptr error in StreamWriter

* Added refreshConnection in JsonStreamWriter to let it handle refreshAppend, flush, and setDescriptor.

* Made JsonStreamWriterOnSchemaUpdateRunnable a private inner class and made refreshConnection not public

* Rerun

* remove import *
  • Loading branch information
allenc3 committed Aug 7, 2020
1 parent 2353b56 commit b9e6cda
Show file tree
Hide file tree
Showing 8 changed files with 1,716 additions and 32 deletions.
Expand Up @@ -15,7 +15,9 @@
*/
package com.google.cloud.bigquery.storage.v1alpha2;

import com.google.api.core.*;
import com.google.api.core.ApiFunction;
import com.google.api.core.ApiFuture;
import com.google.api.core.ApiFutures;
import com.google.api.gax.grpc.GrpcStatusCode;
import com.google.api.gax.rpc.InvalidArgumentException;
import com.google.cloud.bigquery.storage.v1alpha2.ProtoBufProto.ProtoRows;
Expand Down

0 comments on commit b9e6cda

Please sign in to comment.