Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add AppendRowsStream helper to append rows with a BigQueryWriteClient #284

Merged
merged 48 commits into from Sep 10, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
c34e0bf
WIP: write client sample
tswast Aug 9, 2021
c9d6338
add sample with nullable types
tswast Aug 11, 2021
dbe2a21
add schema for all supported types
tswast Aug 11, 2021
e5b7b3f
add complex types to code sample
tswast Aug 12, 2021
2d927b1
refactor sample so that it can be tested
tswast Aug 16, 2021
19b746e
make test assertions more thorough
tswast Aug 16, 2021
f6af466
Merge branch 'master' into b195450856-write-client
tswast Aug 16, 2021
483b0e3
fix lint error
tswast Aug 16, 2021
a74e7ea
Merge remote-tracking branch 'origin/b195450856-write-client' into b1…
tswast Aug 16, 2021
eba8539
remove done TODO
tswast Aug 16, 2021
40e2800
address reviewer comments
tswast Aug 17, 2021
49c3c58
fix tag mismatch
tswast Aug 17, 2021
6d30337
test on multiple regions
tswast Aug 17, 2021
cc969eb
Merge remote-tracking branch 'upstream/master' into b195450856-write-…
tswast Aug 19, 2021
bba6df4
correct comments about why offset exists
tswast Aug 19, 2021
dcc648d
upgrade g-c-b
tswast Aug 19, 2021
12a82a1
WIP: invert stream using BiDi class
tswast Aug 20, 2021
1beb0b9
WIP: attempt to use Future for send instead
tswast Aug 20, 2021
34e5cbd
WIP: use futures, populated by background consumer
tswast Aug 23, 2021
d2491b1
make sure stream is actually open before returning from open
tswast Aug 23, 2021
daf42a8
copy close implementation from pub/sub
tswast Aug 24, 2021
c40b51f
support extra metadata
tswast Aug 25, 2021
8bcc36c
Merge remote-tracking branch 'upstream/master' into b195450856-write-…
tswast Aug 26, 2021
350628d
process exceptions, add open timeout
tswast Aug 26, 2021
4086445
sort imports
tswast Aug 26, 2021
4c7f60e
WIP: unit tests
tswast Aug 26, 2021
f42d871
drain futures when stream closes
tswast Aug 27, 2021
39cd7af
update docs
tswast Aug 27, 2021
cc9a0fb
add callbacks to detect when a stream fails
tswast Aug 27, 2021
1d7f76a
add unit tests
tswast Aug 27, 2021
b054fce
Merge branch 'main' into b195450856-write-client-BiDi
tswast Aug 27, 2021
29e25ce
Merge branch 'main' into b195450856-write-client-BiDi
tswast Aug 31, 2021
faaee9e
add sleep to loop waiting for RPC to be active
tswast Aug 31, 2021
46dd60e
don't freeze if initial RPC fails
tswast Aug 31, 2021
4563e4b
Merge remote-tracking branch 'upstream/main' into b195450856-write-cl…
tswast Sep 3, 2021
22b7b0c
add needed initializations so done() functions
tswast Sep 3, 2021
e307c8f
fail fast when there is a problem with the initial request
tswast Sep 3, 2021
2b26b3d
don't inherit concurrent.futures
tswast Sep 3, 2021
0e91a6a
Merge remote-tracking branch 'upstream/main' into b195450856-write-cl…
tswast Sep 8, 2021
a3185e9
add unit test for open timeout
tswast Sep 8, 2021
61fa79f
🦉 Updates from OwlBot
gcf-owl-bot[bot] Sep 8, 2021
6fc1c75
add manual client to docs
tswast Sep 8, 2021
7bddd4f
Merge remote-tracking branch 'origin/b195450856-write-client-BiDi' in…
tswast Sep 8, 2021
0c5d08c
typo in sample comments
tswast Sep 9, 2021
dc33c88
force timeout and metadata to be kwargs
tswast Sep 9, 2021
9b229a3
unify interface for sending row data
tswast Sep 9, 2021
001a922
pull stream name from merged request
tswast Sep 9, 2021
68507d8
require newer proto-plus for copy_from method
tswast Sep 10, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/bigquery_storage_v1beta2/library.rst
Expand Up @@ -4,3 +4,7 @@ Bigquery Storage v1beta2 API Library
.. automodule:: google.cloud.bigquery_storage_v1beta2.client
:members:
:inherited-members:

.. automodule:: google.cloud.bigquery_storage_v1beta2.writer
:members:
:inherited-members:
5 changes: 5 additions & 0 deletions google/cloud/bigquery_storage_v1beta2/__init__.py
Expand Up @@ -30,10 +30,15 @@ class BigQueryReadClient(client.BigQueryReadClient):
__doc__ = client.BigQueryReadClient.__doc__


class BigQueryWriteClient(client.BigQueryWriteClient):
__doc__ = client.BigQueryWriteClient.__doc__


__all__ = (
# google.cloud.bigquery_storage_v1beta2
"__version__",
"types",
# google.cloud.bigquery_storage_v1beta2.client
"BigQueryReadClient",
"BigQueryWriteClient",
)
12 changes: 9 additions & 3 deletions google/cloud/bigquery_storage_v1beta2/client.py
Expand Up @@ -19,12 +19,14 @@
This is the base from which all interactions with the API occur.
"""

from __future__ import absolute_import

import google.api_core.gapic_v1.method
import google.api_core.retry

from google.cloud.bigquery_storage_v1 import reader
from google.cloud.bigquery_storage_v1beta2.services import big_query_read
from google.cloud.bigquery_storage_v1beta2.services import (
big_query_read,
big_query_write,
)


_SCOPES = (
Expand Down Expand Up @@ -135,3 +137,7 @@ def read_rows(
offset,
{"retry": retry, "timeout": timeout, "metadata": metadata},
)


class BigQueryWriteClient(big_query_write.BigQueryWriteClient):
__doc__ = big_query_write.BigQueryWriteClient.__doc__
17 changes: 17 additions & 0 deletions google/cloud/bigquery_storage_v1beta2/exceptions.py
@@ -0,0 +1,17 @@
# Copyright 2021 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


class StreamClosedError(Exception):
"""Operation not supported while stream is closed."""