Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
feat: add BigQueryWriteClient where append_rows returns a helper …
…for writing rows (#284)

* WIP: write client sample

* add sample with nullable types

* add schema for all supported types

* add complex types to code sample

* refactor sample so that it can be tested

* make test assertions more thorough

* fix lint error

* remove done TODO

* address reviewer comments

* fix tag mismatch

* test on multiple regions

* correct comments about why offset exists

* upgrade g-c-b

* WIP: invert stream using BiDi class

* WIP: attempt to use Future for send instead

* WIP: use futures, populated by background consumer

* make sure stream is actually open before returning from open

* copy close implementation from pub/sub

* support extra metadata

* process exceptions, add open timeout

* sort imports

* WIP: unit tests

* drain futures when stream closes

* update docs

* add callbacks to detect when a stream fails

* add unit tests

* add sleep to loop waiting for RPC to be active

* don't freeze if initial RPC fails

* add needed initializations so done() functions

* fail fast when there is a problem with the initial request

* don't inherit concurrent.futures

It's unnecessary and kept resulting in stuff getting stuck.

* add unit test for open timeout

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

* add manual client to docs

* typo in sample comments

* force timeout and metadata to be kwargs

* unify interface for sending row data

* pull stream name from merged request

* require newer proto-plus for copy_from method

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
  • Loading branch information
tswast and gcf-owl-bot[bot] committed Sep 10, 2021
1 parent c7ac698 commit 2461f63
Show file tree
Hide file tree
Showing 24 changed files with 1,943 additions and 41 deletions.
4 changes: 4 additions & 0 deletions docs/bigquery_storage_v1beta2/library.rst
Expand Up @@ -4,3 +4,7 @@ Bigquery Storage v1beta2 API Library
.. automodule:: google.cloud.bigquery_storage_v1beta2.client
:members:
:inherited-members:

.. automodule:: google.cloud.bigquery_storage_v1beta2.writer
:members:
:inherited-members:
5 changes: 5 additions & 0 deletions google/cloud/bigquery_storage_v1beta2/__init__.py
Expand Up @@ -30,10 +30,15 @@ class BigQueryReadClient(client.BigQueryReadClient):
__doc__ = client.BigQueryReadClient.__doc__


class BigQueryWriteClient(client.BigQueryWriteClient):
__doc__ = client.BigQueryWriteClient.__doc__


__all__ = (
# google.cloud.bigquery_storage_v1beta2
"__version__",
"types",
# google.cloud.bigquery_storage_v1beta2.client
"BigQueryReadClient",
"BigQueryWriteClient",
)
12 changes: 9 additions & 3 deletions google/cloud/bigquery_storage_v1beta2/client.py
Expand Up @@ -19,12 +19,14 @@
This is the base from which all interactions with the API occur.
"""

from __future__ import absolute_import

import google.api_core.gapic_v1.method
import google.api_core.retry

from google.cloud.bigquery_storage_v1 import reader
from google.cloud.bigquery_storage_v1beta2.services import big_query_read
from google.cloud.bigquery_storage_v1beta2.services import (
big_query_read,
big_query_write,
)


_SCOPES = (
Expand Down Expand Up @@ -135,3 +137,7 @@ def read_rows(
offset,
{"retry": retry, "timeout": timeout, "metadata": metadata},
)


class BigQueryWriteClient(big_query_write.BigQueryWriteClient):
__doc__ = big_query_write.BigQueryWriteClient.__doc__
17 changes: 17 additions & 0 deletions google/cloud/bigquery_storage_v1beta2/exceptions.py
@@ -0,0 +1,17 @@
# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


class StreamClosedError(Exception):
"""Operation not supported while stream is closed."""

0 comments on commit 2461f63

Please sign in to comment.