feat(v1): add batch write (#62)

* Integrate Python GAPIC Microgenerator in googleapis. This PR uses using documentai as an example. Depends on googleapis/gapic-generator-python#402

PiperOrigin-RevId: 309824146

Source-Author: Google APIs <noreply@google.com>
Source-Date: Mon May 4 15:06:44 2020 -0700
Source-Repo: googleapis/googleapis
Source-Sha: e0f9d9e1f9de890db765be46f45ca8490723e3eb
Source-Link: googleapis/googleapis@e0f9d9e

* chore(python): create thread_safety doc

This will be included in index.rst files. See https://github.com/googleapis/python-bigtable/pull/26/files

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Tue May 5 10:06:52 2020 -0700
Source-Repo: googleapis/synthtool
Source-Sha: 09c48461232ce929c34386259eb59018ad2d8eef
Source-Link: googleapis/synthtool@09c4846

* chore(python): remove extra #!/bin/bash

Co-authored-by: Jeffrey Rennie <rennie@google.com>

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Tue May 12 09:51:12 2020 -0700
Source-Repo: googleapis/synthtool
Source-Sha: 7482e79a82e353248769d819788adc1213e8c207
Source-Link: googleapis/synthtool@7482e79

* fix: avoid running out of disk space

by cleaning up after each inner synth run

Source-Author: Jeffrey Rennie <rennie@google.com>
Source-Date: Fri May 15 10:53:36 2020 -0700
Source-Repo: googleapis/synthtool
Source-Sha: fe8bbfdf2fef3bed8ebd461dcd33dfdd28604d02
Source-Link: googleapis/synthtool@fe8bbfd

* fix: restore GAPIC v2 retry configs

Restores GAPIC v2 retry config overrides that were
mistakenly removed during migration.

PiperOrigin-RevId: 312088359

Source-Author: Google APIs <noreply@google.com>
Source-Date: Mon May 18 08:57:12 2020 -0700
Source-Repo: googleapis/googleapis
Source-Sha: 5a90d467aa65e7f038f87585e8fbb45d74475e7c
Source-Link: googleapis/googleapis@5a90d46

* chore(python): remove one remaining extra bash line

To avoid generating bogus PRs

Co-authored-by: Jeffrey Rennie <rennie@google.com>

Source-Author: Carlos de la Guardia <cguardia@yahoo.com>
Source-Date: Tue May 19 11:57:19 2020 -0500
Source-Repo: googleapis/synthtool
Source-Sha: d2364eb80b840a36136c8ce12f1c6efabcc9600e
Source-Link: googleapis/synthtool@d2364eb

* fix: bigtable v2 retry config settings to disable streaming RPC retries

PiperOrigin-RevId: 312558019

Source-Author: Google APIs <noreply@google.com>
Source-Date: Wed May 20 14:40:11 2020 -0700
Source-Repo: googleapis/googleapis
Source-Sha: a175708acd7a367ab3ed120fa562331e1761d825
Source-Link: googleapis/googleapis@a175708

* Upgrade protoc-docs-plugin version to 0.7.0 to fix unstable docstrings.

PiperOrigin-RevId: 312689208

Source-Author: Google APIs <noreply@google.com>
Source-Date: Thu May 21 10:00:47 2020 -0700
Source-Repo: googleapis/googleapis
Source-Sha: dec3204175104cef49bf21d685d5517caaf0058f
Source-Link: googleapis/googleapis@dec3204

* bazel: update protobuf, rules_go, gazelle, and gapic-generator-go versions - protobuf v3.12.1 - rules_go v0.23.0 - gazelle v0.21.0 - gapic-generator-go v0.14.1

PiperOrigin-RevId: 313460921

Source-Author: Google APIs <noreply@google.com>
Source-Date: Wed May 27 14:10:16 2020 -0700
Source-Repo: googleapis/googleapis
Source-Sha: c4e37010d74071851ff24121f522e802231ac86e
Source-Link: googleapis/googleapis@c4e3701

* docs: update python docs template

* docs: update python docs template

* sphinx change

Source-Author: kolea2 <45548808+kolea2@users.noreply.github.com>
Source-Date: Wed May 27 20:44:34 2020 -0400
Source-Repo: googleapis/synthtool
Source-Sha: 71b8a272549c06b5768d00fa48d3ae990e871bec
Source-Link: googleapis/synthtool@71b8a27

* feat: add templates for python samples projects

These templates will be used for templates in python-docs-samples
and in Python client libraries. The README generation code is a modified
version of
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/scripts/readme-gen.

Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Thu May 28 14:39:58 2020 -0700
Source-Repo: googleapis/synthtool
Source-Sha: ffe10407ee2f261c799fb0d01bf32a8abc67ed1e
Source-Link: googleapis/synthtool@ffe1040

* chore: update default ignores for Python libraries

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Thu May 28 14:43:37 2020 -0700
Source-Repo: googleapis/synthtool
Source-Sha: e99975b6b49827b8720f0a885e218dbdb67849ca
Source-Link: googleapis/synthtool@e99975b

* feat: allow custom python versions in noxfile

Libraries on the microgenerator support a smaller range of Python versions (3.6+).

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Thu May 28 18:22:04 2020 -0700
Source-Repo: googleapis/synthtool
Source-Sha: 4e1d2cb79b02d7496b1452f91c518630c207145e
Source-Link: googleapis/synthtool@4e1d2cb

* chore: update gapic-generator and go microgen

changes include:
- build_gen: go lro gapic used as dep
- go_gapic_library: fixes shading of go_library importpath

PiperOrigin-RevId: 314363155

Source-Author: Google APIs <noreply@google.com>
Source-Date: Tue Jun 2 10:56:09 2020 -0700
Source-Repo: googleapis/googleapis
Source-Sha: 3a4894c4f0da3e763aca2c67bd280ae915177450
Source-Link: googleapis/googleapis@3a4894c

* firestore: add BatchWrite

PiperOrigin-RevId: 314471006

Source-Author: Google APIs <noreply@google.com>
Source-Date: Tue Jun 2 22:32:52 2020 -0700
Source-Repo: googleapis/googleapis
Source-Sha: d5fe42c39cd35f95131a0267314ae108ab1bef8d
Source-Link: googleapis/googleapis@d5fe42c

* chore: set Ruby namespace in proto options

PiperOrigin-RevId: 316123861

Source-Author: Google APIs <noreply@google.com>
Source-Date: Fri Jun 12 10:03:16 2020 -0700
Source-Repo: googleapis/googleapis
Source-Sha: 66f959b51c255dd6bef79ef221865e53a50df878
Source-Link: googleapis/googleapis@66f959b

* fix: use protoc-docs-plugin 0.8.0

Fixes issue with missing newline before 'Attributes' in Python docstrings.

PiperOrigin-RevId: 316182409

Source-Author: Google APIs <noreply@google.com>
Source-Date: Fri Jun 12 14:52:11 2020 -0700
Source-Repo: googleapis/googleapis
Source-Sha: 184661793fbe3b89f2b485c303e7466cef9d21a1
Source-Link: googleapis/googleapis@1846617

* feat: retry CommitRequests that fail with UNAVAILABLE

PiperOrigin-RevId: 316514901

Source-Author: Google APIs <noreply@google.com>
Source-Date: Mon Jun 15 12:00:19 2020 -0700
Source-Repo: googleapis/googleapis
Source-Sha: 9b170a4f3beb16f7dab513520a07be48fcb88670
Source-Link: googleapis/googleapis@9b170a4

* fix(python): change autodoc_default_flags to autodoc_default_options

Source-Author: HemangChothani <50404902+HemangChothani@users.noreply.github.com>
Source-Date: Thu Jun 18 22:50:02 2020 +0530
Source-Repo: googleapis/synthtool
Source-Sha: cd522c3b4dde821766d95c80ae5aeb43d7a41170
Source-Link: googleapis/synthtool@cd522c3

* feat(python): add defaults for microgenerator libraries

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Mon Jun 22 17:56:27 2020 -0700
Source-Repo: googleapis/synthtool
Source-Sha: db4f15fedc76e04018e6e43ec403e804e1142158
Source-Link: googleapis/synthtool@db4f15f

* chore(python): fix line breaks in template and use newer black version

* chore(python): fix line breaks in template

* fix: also update black version

Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Source-Date: Tue Jun 23 02:33:35 2020 -0700
Source-Repo: googleapis/synthtool
Source-Sha: cf2eff09d0f5319a4dc5cdce2b6356d85af4a798
Source-Link: googleapis/synthtool@cf2eff0

* chore: manual regen

* fix: more synth replaces

* fix: more synth replaces

* fix: fix the noxfile

Co-authored-by: Bu Sun Kim <busunkim@google.com>
Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>

.flake8

@@ -21,6 +21,8 @@ exclude =
   # Exclude generated code.
   **/proto/**
   **/gapic/**
+  **/services/**
+  **/types/**
   *_pb2.py
 
   # Standard linting exemptions.

.gitignore

@@ -10,6 +10,7 @@
 dist
 build
 eggs
+.eggs
 parts
 bin
 var
@@ -49,6 +50,7 @@ bigquery/docs/generated
 # Virtual environment
 env/
 coverage.xml
+sponge_log.xml
 
 # System test environment variables.
 system_tests/local_test_setup

.kokoro/publish-docs.sh

@@ -13,8 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-#!/bin/bash
-
 set -eo pipefail
 
 # Disable buffering, so that the logs stream through.

.kokoro/release.sh

@@ -13,8 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-#!/bin/bash
-
 set -eo pipefail
 
 # Start the releasetool reporter

.kokoro/samples/lint/common.cfg

@@ -0,0 +1,34 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Build logs will be here
+action {
+  define_artifacts {
+    regex: "**/*sponge_log.xml"
+  }
+}
+
+# Specify which tests to run
+env_vars: {
+    key: "RUN_TESTS_SESSION"
+    value: "lint"
+}
+
+env_vars: {
+    key: "TRAMPOLINE_BUILD_FILE"
+    value: "github/python-firestore/.kokoro/test-samples.sh"
+}
+
+# Configure the docker image for kokoro-trampoline.
+env_vars: {
+    key: "TRAMPOLINE_IMAGE"
+    value: "gcr.io/cloud-devrel-kokoro-resources/python-samples-testing-docker"
+}
+
+# Download secrets for samples
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/python-docs-samples"
+
+# Download trampoline resources.
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/trampoline"
+
+# Use the trampoline script to run in docker.
+build_file: "python-firestore/.kokoro/trampoline.sh"

.kokoro/samples/lint/continuous.cfg

@@ -0,0 +1,6 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "INSTALL_LIBRARY_FROM_SOURCE"
+    value: "True"
+}

.kokoro/samples/lint/periodic.cfg

@@ -0,0 +1,6 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "INSTALL_LIBRARY_FROM_SOURCE"
+    value: "False"
+}

.kokoro/samples/lint/presubmit.cfg

@@ -0,0 +1,6 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "INSTALL_LIBRARY_FROM_SOURCE"
+    value: "True"
+}

.kokoro/samples/python3.6/common.cfg

@@ -0,0 +1,34 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Build logs will be here
+action {
+  define_artifacts {
+    regex: "**/*sponge_log.xml"
+  }
+}
+
+# Specify which tests to run
+env_vars: {
+    key: "RUN_TESTS_SESSION"
+    value: "py-3.6"
+}
+
+env_vars: {
+    key: "TRAMPOLINE_BUILD_FILE"
+    value: "github/python-firestore/.kokoro/test-samples.sh"
+}
+
+# Configure the docker image for kokoro-trampoline.
+env_vars: {
+    key: "TRAMPOLINE_IMAGE"
+    value: "gcr.io/cloud-devrel-kokoro-resources/python-samples-testing-docker"
+}
+
+# Download secrets for samples
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/python-docs-samples"
+
+# Download trampoline resources.
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/trampoline"
+
+# Use the trampoline script to run in docker.
+build_file: "python-firestore/.kokoro/trampoline.sh"

.kokoro/samples/python3.6/continuous.cfg

@@ -0,0 +1,7 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "INSTALL_LIBRARY_FROM_SOURCE"
+    value: "True"
+}
+

.kokoro/samples/python3.6/periodic.cfg

@@ -0,0 +1,6 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "INSTALL_LIBRARY_FROM_SOURCE"
+    value: "False"
+}

.kokoro/samples/python3.6/presubmit.cfg

@@ -0,0 +1,6 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "INSTALL_LIBRARY_FROM_SOURCE"
+    value: "True"
+}

.kokoro/samples/python3.7/common.cfg

@@ -0,0 +1,34 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Build logs will be here
+action {
+  define_artifacts {
+    regex: "**/*sponge_log.xml"
+  }
+}
+
+# Specify which tests to run
+env_vars: {
+    key: "RUN_TESTS_SESSION"
+    value: "py-3.7"
+}
+
+env_vars: {
+    key: "TRAMPOLINE_BUILD_FILE"
+    value: "github/python-firestore/.kokoro/test-samples.sh"
+}
+
+# Configure the docker image for kokoro-trampoline.
+env_vars: {
+    key: "TRAMPOLINE_IMAGE"
+    value: "gcr.io/cloud-devrel-kokoro-resources/python-samples-testing-docker"
+}
+
+# Download secrets for samples
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/python-docs-samples"
+
+# Download trampoline resources.
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/trampoline"
+
+# Use the trampoline script to run in docker.
+build_file: "python-firestore/.kokoro/trampoline.sh"

.kokoro/samples/python3.7/continuous.cfg

@@ -0,0 +1,6 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "INSTALL_LIBRARY_FROM_SOURCE"
+    value: "True"
+}

.kokoro/samples/python3.7/periodic.cfg

@@ -0,0 +1,6 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "INSTALL_LIBRARY_FROM_SOURCE"
+    value: "False"
+}

.kokoro/samples/python3.7/presubmit.cfg

@@ -0,0 +1,6 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "INSTALL_LIBRARY_FROM_SOURCE"
+    value: "True"
+}

.kokoro/samples/python3.8/common.cfg

@@ -0,0 +1,34 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+# Build logs will be here
+action {
+  define_artifacts {
+    regex: "**/*sponge_log.xml"
+  }
+}
+
+# Specify which tests to run
+env_vars: {
+    key: "RUN_TESTS_SESSION"
+    value: "py-3.8"
+}
+
+env_vars: {
+    key: "TRAMPOLINE_BUILD_FILE"
+    value: "github/python-firestore/.kokoro/test-samples.sh"
+}
+
+# Configure the docker image for kokoro-trampoline.
+env_vars: {
+    key: "TRAMPOLINE_IMAGE"
+    value: "gcr.io/cloud-devrel-kokoro-resources/python-samples-testing-docker"
+}
+
+# Download secrets for samples
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/python-docs-samples"
+
+# Download trampoline resources.
+gfile_resources: "/bigstore/cloud-devrel-kokoro-resources/trampoline"
+
+# Use the trampoline script to run in docker.
+build_file: "python-firestore/.kokoro/trampoline.sh"

.kokoro/samples/python3.8/continuous.cfg

@@ -0,0 +1,6 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "INSTALL_LIBRARY_FROM_SOURCE"
+    value: "True"
+}

.kokoro/samples/python3.8/periodic.cfg

@@ -0,0 +1,6 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "INSTALL_LIBRARY_FROM_SOURCE"
+    value: "False"
+}

.kokoro/samples/python3.8/presubmit.cfg

@@ -0,0 +1,6 @@
+# Format: //devtools/kokoro/config/proto/build.proto
+
+env_vars: {
+    key: "INSTALL_LIBRARY_FROM_SOURCE"
+    value: "True"
+}

.kokoro/test-samples.sh

@@ -0,0 +1,104 @@
+#!/bin/bash
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+# `-e` enables the script to automatically fail when a command fails
+# `-o pipefail` sets the exit code to the rightmost comment to exit with a non-zero
+set -eo pipefail
+# Enables `**` to include files nested inside sub-folders
+shopt -s globstar
+
+cd github/python-firestore
+
+# Run periodic samples tests at latest release
+if [[ $KOKORO_BUILD_ARTIFACTS_SUBDIR = *"periodic"* ]]; then
+    LATEST_RELEASE=$(git describe --abbrev=0 --tags)
+    git checkout $LATEST_RELEASE
+fi
+
+# Disable buffering, so that the logs stream through.
+export PYTHONUNBUFFERED=1
+
+# Debug: show build environment
+env | grep KOKORO
+
+# Install nox
+python3.6 -m pip install --upgrade --quiet nox
+
+# Use secrets acessor service account to get secrets
+if [[ -f "${KOKORO_GFILE_DIR}/secrets_viewer_service_account.json" ]]; then
+    gcloud auth activate-service-account \
+	   --key-file="${KOKORO_GFILE_DIR}/secrets_viewer_service_account.json" \
+	   --project="cloud-devrel-kokoro-resources"
+fi
+
+# This script will create 3 files:
+# - testing/test-env.sh
+# - testing/service-account.json
+# - testing/client-secrets.json
+./scripts/decrypt-secrets.sh
+
+source ./testing/test-env.sh
+export GOOGLE_APPLICATION_CREDENTIALS=$(pwd)/testing/service-account.json
+
+# For cloud-run session, we activate the service account for gcloud sdk.
+gcloud auth activate-service-account \
+       --key-file "${GOOGLE_APPLICATION_CREDENTIALS}"
+
+export GOOGLE_CLIENT_SECRETS=$(pwd)/testing/client-secrets.json
+
+echo -e "\n******************** TESTING PROJECTS ********************"
+
+# Switch to 'fail at end' to allow all tests to complete before exiting.
+set +e
+# Use RTN to return a non-zero value if the test fails.
+RTN=0
+ROOT=$(pwd)
+# Find all requirements.txt in the samples directory (may break on whitespace).
+for file in samples/**/requirements.txt; do
+    cd "$ROOT"
+    # Navigate to the project folder.
+    file=$(dirname "$file")
+    cd "$file"
+
+    echo "------------------------------------------------------------"
+    echo "- testing $file"
+    echo "------------------------------------------------------------"
+
+    # Use nox to execute the tests for the project.
+    python3.6 -m nox -s "$RUN_TESTS_SESSION"
+    EXIT=$?
+
+    # If this is a periodic build, send the test log to the Build Cop Bot.
+    # See https://github.com/googleapis/repo-automation-bots/tree/master/packages/buildcop.
+    if [[ $KOKORO_BUILD_ARTIFACTS_SUBDIR = *"periodic"* ]]; then
+      chmod +x $KOKORO_GFILE_DIR/linux_amd64/buildcop
+      $KOKORO_GFILE_DIR/linux_amd64/buildcop
+    fi
+
+    if [[ $EXIT -ne 0 ]]; then
+      RTN=1
+      echo -e "\n Testing failed: Nox returned a non-zero exit code. \n"
+    else
+      echo -e "\n Testing completed.\n"
+    fi
+
+done
+cd "$ROOT"
+
+# Workaround for Kokoro permissions issue: delete secrets
+rm testing/{test-env.sh,client-secrets.json,service-account.json}
+
+exit "$RTN"

MANIFEST.in

@@ -20,3 +20,6 @@ recursive-include google *.json *.proto
 recursive-include tests *
 global-exclude *.py[co]
 global-exclude __pycache__
+
+# Exclude scripts for samples readmegen
+prune scripts/readme-gen