feat!: migrate to v2.0.0 (#147)

* chore: remove old generated files

* chore: remove type files

* refactor: move handwritten files to google/cloud/spanner

* chore: update synth.py

* chore: regen library via microgenerator

* fix: update imports to use new files

* fix: update _helper.py

* update backup.py

* fix: update batch.py

* fix: update param_types.py

* fix: update pool.py

* test: update test_keyset.py

* test: update test__opentelemetry_tracing.py

* fix: update client.py

* test: update test_client.py

* fix: update database.py

* test: update test_database.py

* fix: update commmit call missing kwarg

* test: update test_session.py

* fix: update streamed.py

* test: update TestStreamedResultSet tests in streamed.py

* fix: update instance.py

* test: update test_instance.py

* fix: add _parse_value method to _helper.py

* test: update test/_helper.py

* fix: update transaction.py and test_transaction.py

* fix: remove creating list_value_pb

* fix: update snapshot.py and test_snapshot.py

* fix: update setup.py

* fix: use struct pb for partition_query

* fix: support merging RepeatedComposite objects

* fix: replace dict with Statements

* fix: replace dict with Statements

* lint: format code

* fix: update synth.py

* chore: regen with gapic-generator v0.33.4

* test: increase test coverage

* fix: use backup pb instead of dict

* fix: update api calls to use supported kwargs

* fix: update api calls to use supported kwargs and remove unused function

* test: update system tests

* test: add tests for _parse_value function

* test: update empty list Value to correctly reflect the format

* refactor: use _parse_value in _parse_value_pb

* refactor: remove unneeded wrapper classes

* fix: use default retry (see golang) as ExecuteStreamingSql does not define retry settings

* refactor: remove unneeded wrapper class

* fix: use pb for params kwargs

* test: increase coverage

* test: correctly assert UNKNOWN error

* chore: remove unneeded replacesments and regen with gapic-generator v0.33.6

* fix: update kwarg to type_

* refactor: remove unused imports

* refactor: remove unused imports

* fix: update kwarg to all_

* fix: update kwarg to type_

* fix: update kwarg and attribute to type_

* fix: update kwarg to type_

* test: fix test name

* style: format code

* fix: update kwarg to type_

* fix: update api calls

* test: update param types to use param_types types

* test: update numeric tests

* test: remove unused variable

* refactor: remove unused import

* docs: update doc references

* test: increase test coverage

* chore: test with and without opentelemetry for full test coverage

* chore: regen samples README

* fix: update emulator Github action

* docs: manually fix typo that breaks docs

* docs: remove unsupported markdown and point to link with removed info

* docs: fix broken hyperlink

* chore: add replacement for docs formatting issue

* chore: regen library (via synth)

* chore: exclude noxfile.py from being generated

* refator: move handwritten files back into google/cloud/spanner_v1

* style: fix lint

* fix: update sample to use correct type

* fix: create alias for proto types

* fix: update imports

* test: update test to use proto-plus structures

* ci: update python version for emulator tests

* fix: update backup pagination example

* test: revert test_keyset for coverage

* test: fix expected ranges

* fix: ignore google/cloud/spanner_v1/__init__.py for regen

* chore: regen (via synth)

* docs: revert import changes

* refactor: address comments

* docs: add UPDATING guide

* refactor: revert imports

* feat: remove deprecated arguments

* docs: update guide to mention deprecated arguments

* test: lower required coverage to 99%

* test: remove deprecated options

* style: fix lint error

* Update UPGRADING.md

Co-authored-by: skuruppu <skuruppu@google.com>

Co-authored-by: larkee <larkee@users.noreply.github.com>
Co-authored-by: skuruppu <skuruppu@google.com>

.github/workflows/integration-tests-against-emulator.yaml

@@ -21,11 +21,11 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v2
         with:
-          python-version: 3.7
+          python-version: 3.8
       - name: Install nox
         run: python -m pip install nox
       - name: Run system tests
-        run: nox -s system-3.7
+        run: nox -s system
         env:
           SPANNER_EMULATOR_HOST: localhost:9010
           GOOGLE_CLOUD_PROJECT: emulator-test-project

.kokoro/docker/docs/Dockerfile

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from ubuntu:20.10
+from ubuntu:20.04
 
 ENV DEBIAN_FRONTEND noninteractive
 

.kokoro/docs/common.cfg

@@ -30,7 +30,7 @@ env_vars: {
 
 env_vars: {
     key: "V2_STAGING_BUCKET"
-    value: "docs-staging-v2-staging"
+    value: "docs-staging-v2"
 }
 
 # It will upload the docker image after successful builds.

.kokoro/populate-secrets.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+# Copyright 2020 Google LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eo pipefail
+
+function now { date +"%Y-%m-%d %H:%M:%S" | tr -d '\n' ;}
+function msg { println "$*" >&2 ;}
+function println { printf '%s\n' "$(now) $*" ;}
+
+
+# Populates requested secrets set in SECRET_MANAGER_KEYS from service account:
+# kokoro-trampoline@cloud-devrel-kokoro-resources.iam.gserviceaccount.com
+SECRET_LOCATION="${KOKORO_GFILE_DIR}/secret_manager"
+msg "Creating folder on disk for secrets: ${SECRET_LOCATION}"
+mkdir -p ${SECRET_LOCATION}
+for key in $(echo ${SECRET_MANAGER_KEYS} | sed "s/,/ /g")
+do
+  msg "Retrieving secret ${key}"
+  docker run --entrypoint=gcloud \
+    --volume=${KOKORO_GFILE_DIR}:${KOKORO_GFILE_DIR} \
+    gcr.io/google.com/cloudsdktool/cloud-sdk \
+    secrets versions access latest \
+    --project cloud-devrel-kokoro-resources \
+    --secret ${key} > \
+    "${SECRET_LOCATION}/${key}"
+  if [[ $? == 0 ]]; then
+    msg "Secret written to ${SECRET_LOCATION}/${key}"
+  else
+    msg "Error retrieving secret ${key}"
+  fi
+done

.kokoro/release/common.cfg

@@ -23,42 +23,18 @@ env_vars: {
     value: "github/python-spanner/.kokoro/release.sh"
 }
 
-# Fetch the token needed for reporting release status to GitHub
-before_action {
-  fetch_keystore {
-    keystore_resource {
-      keystore_config_id: 73713
-      keyname: "yoshi-automation-github-key"
-    }
-  }
-}
-
-# Fetch PyPI password
-before_action {
-  fetch_keystore {
-    keystore_resource {
-      keystore_config_id: 73713
-      keyname: "google_cloud_pypi_password"
-    }
-  }
-}
-
-# Fetch magictoken to use with Magic Github Proxy 
-before_action {
-  fetch_keystore {
-    keystore_resource {
-      keystore_config_id: 73713
-      keyname: "releasetool-magictoken"
-    }
-  }
+# Fetch PyPI password	
+before_action {	
+  fetch_keystore {	
+    keystore_resource {	
+      keystore_config_id: 73713	
+      keyname: "google_cloud_pypi_password"	
+    }	
+  }	
 }
 
-# Fetch api key to use with Magic Github Proxy 
-before_action {
-  fetch_keystore {
-    keystore_resource {
-      keystore_config_id: 73713
-      keyname: "magic-github-proxy-api-key"
-    }
-  }
-}
+# Tokens needed to report release status back to GitHub
+env_vars: {
+  key: "SECRET_MANAGER_KEYS"
+  value: "releasetool-publish-reporter-app,releasetool-publish-reporter-googleapis-installation,releasetool-publish-reporter-pem"
+}

.kokoro/samples/python3.6/common.cfg

@@ -13,6 +13,12 @@ env_vars: {
     value: "py-3.6"
 }
 
+# Declare build specific Cloud project.
+env_vars: {
+    key: "BUILD_SPECIFIC_GCLOUD_PROJECT"
+    value: "python-docs-samples-tests-py36"
+}
+
 env_vars: {
     key: "TRAMPOLINE_BUILD_FILE"
     value: "github/python-spanner/.kokoro/test-samples.sh"

.kokoro/samples/python3.7/common.cfg

@@ -13,6 +13,12 @@ env_vars: {
     value: "py-3.7"
 }
 
+# Declare build specific Cloud project.
+env_vars: {
+    key: "BUILD_SPECIFIC_GCLOUD_PROJECT"
+    value: "python-docs-samples-tests-py37"
+}
+
 env_vars: {
     key: "TRAMPOLINE_BUILD_FILE"
     value: "github/python-spanner/.kokoro/test-samples.sh"

.kokoro/samples/python3.8/common.cfg

@@ -13,6 +13,12 @@ env_vars: {
     value: "py-3.8"
 }
 
+# Declare build specific Cloud project.
+env_vars: {
+    key: "BUILD_SPECIFIC_GCLOUD_PROJECT"
+    value: "python-docs-samples-tests-py38"
+}
+
 env_vars: {
     key: "TRAMPOLINE_BUILD_FILE"
     value: "github/python-spanner/.kokoro/test-samples.sh"

.kokoro/test-samples.sh

@@ -28,6 +28,12 @@ if [[ $KOKORO_BUILD_ARTIFACTS_SUBDIR = *"periodic"* ]]; then
     git checkout $LATEST_RELEASE
 fi
 
+# Exit early if samples directory doesn't exist
+if [ ! -d "./samples" ]; then
+  echo "No tests run. `./samples` not found"
+  exit 0
+fi
+
 # Disable buffering, so that the logs stream through.
 export PYTHONUNBUFFERED=1
 
@@ -101,4 +107,4 @@ cd "$ROOT"
 # Workaround for Kokoro permissions issue: delete secrets
 rm testing/{test-env.sh,client-secrets.json,service-account.json}
 
-exit "$RTN"
+exit "$RTN"

.kokoro/trampoline.sh

@@ -15,9 +15,14 @@
 
 set -eo pipefail
 
-python3 "${KOKORO_GFILE_DIR}/trampoline_v1.py"  || ret_code=$?
+# Always run the cleanup script, regardless of the success of bouncing into
+# the container.
+function cleanup() {
+    chmod +x ${KOKORO_GFILE_DIR}/trampoline_cleanup.sh
+    ${KOKORO_GFILE_DIR}/trampoline_cleanup.sh
+    echo "cleanup";
+}
+trap cleanup EXIT
 
-chmod +x ${KOKORO_GFILE_DIR}/trampoline_cleanup.sh
-${KOKORO_GFILE_DIR}/trampoline_cleanup.sh || true
-
-exit ${ret_code}
+$(dirname $0)/populate-secrets.sh # Secret Manager secrets.
+python3 "${KOKORO_GFILE_DIR}/trampoline_v1.py"

CODE_OF_CONDUCT.md

@@ -1,44 +1,95 @@
 <!-- # Generated by synthtool. DO NOT EDIT! !-->
-# Contributor Code of Conduct
+# Code of Conduct
 
-As contributors and maintainers of this project,
-and in the interest of fostering an open and welcoming community,
-we pledge to respect all people who contribute through reporting issues,
-posting feature requests, updating documentation,
-submitting pull requests or patches, and other activities.
+## Our Pledge
 
-We are committed to making participation in this project
-a harassment-free experience for everyone,
-regardless of level of experience, gender, gender identity and expression,
-sexual orientation, disability, personal appearance,
-body size, race, ethnicity, age, religion, or nationality.
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of
+experience, education, socio-economic status, nationality, personal appearance,
+race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+*   Using welcoming and inclusive language
+*   Being respectful of differing viewpoints and experiences
+*   Gracefully accepting constructive criticism
+*   Focusing on what is best for the community
+*   Showing empathy towards other community members
 
 Examples of unacceptable behavior by participants include:
 
-* The use of sexualized language or imagery
-* Personal attacks
-* Trolling or insulting/derogatory comments
-* Public or private harassment
-* Publishing other's private information,
-such as physical or electronic
-addresses, without explicit permission
-* Other unethical or unprofessional conduct.
+*   The use of sexualized language or imagery and unwelcome sexual attention or
+    advances
+*   Trolling, insulting/derogatory comments, and personal or political attacks
+*   Public or private harassment
+*   Publishing others' private information, such as a physical or electronic
+    address, without explicit permission
+*   Other conduct which could reasonably be considered inappropriate in a
+    professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
 
 Project maintainers have the right and responsibility to remove, edit, or reject
-comments, commits, code, wiki edits, issues, and other contributions
-that are not aligned to this Code of Conduct.
-By adopting this Code of Conduct,
-project maintainers commit themselves to fairly and consistently
-applying these principles to every aspect of managing this project.
-Project maintainers who do not follow or enforce the Code of Conduct
-may be permanently removed from the project team.
-
-This code of conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community.
-
-Instances of abusive, harassing, or otherwise unacceptable behavior
-may be reported by opening an issue
-or contacting one or more of the project maintainers.
-
-This Code of Conduct is adapted from the [Contributor Covenant](http://contributor-covenant.org), version 1.2.0,
-available at [http://contributor-covenant.org/version/1/2/0/](http://contributor-covenant.org/version/1/2/0/)
+comments, commits, code, wiki edits, issues, and other contributions that are
+not aligned to this Code of Conduct, or to ban temporarily or permanently any
+contributor for other behaviors that they deem inappropriate, threatening,
+offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+This Code of Conduct also applies outside the project spaces when the Project
+Steward has a reasonable belief that an individual's behavior may have a
+negative impact on the project or its community.
+
+## Conflict Resolution
+
+We do not believe that all conflict is bad; healthy debate and disagreement
+often yield positive results. However, it is never okay to be disrespectful or
+to engage in behavior that violates the project’s code of conduct.
+
+If you see someone violating the code of conduct, you are encouraged to address
+the behavior directly with those involved. Many issues can be resolved quickly
+and easily, and this gives people more control over the outcome of their
+dispute. If you are unable to resolve the matter for any reason, or if the
+behavior is threatening or harassing, report it. We are dedicated to providing
+an environment where participants feel welcome and safe.
+
+
+Reports should be directed to *googleapis-stewards@google.com*, the
+Project Steward(s) for *Google Cloud Client Libraries*. It is the Project Steward’s duty to
+receive and address reported violations of the code of conduct. They will then
+work with a committee consisting of representatives from the Open Source
+Programs Office and the Google Open Source Strategy team. If for any reason you
+are uncomfortable reaching out to the Project Steward, please email
+opensource@google.com.
+
+We will investigate every complaint, but you may not receive a direct response.
+We will use our discretion in determining when and how to follow up on reported
+incidents, which may range from not taking action to permanent expulsion from
+the project and project-sponsored spaces. We will notify the accused of the
+report and provide them an opportunity to discuss it before any action is taken.
+The identity of the reporter will be omitted from the details of the report
+supplied to the accused. In potentially harmful situations, such as ongoing
+harassment or threats to anyone's safety, we may take action without notice.
+
+## Attribution
+
+This Code of Conduct is adapted from the Contributor Covenant, version 1.4,
+available at
+https://www.contributor-covenant.org/version/1/4/code-of-conduct.html

CONTRIBUTING.rst

@@ -80,25 +80,6 @@ We use `nox <https://nox.readthedocs.io/en/latest/>`__ to instrument our tests.
 
 .. nox: https://pypi.org/project/nox/
 
-Note on Editable Installs / Develop Mode
-========================================
-
-- As mentioned previously, using ``setuptools`` in `develop mode`_
-  or a ``pip`` `editable install`_ is not possible with this
-  library. This is because this library uses `namespace packages`_.
-  For context see `Issue #2316`_ and the relevant `PyPA issue`_.
-
-  Since ``editable`` / ``develop`` mode can't be used, packages
-  need to be installed directly. Hence your changes to the source
-  tree don't get incorporated into the **already installed**
-  package.
-
-.. _namespace packages: https://www.python.org/dev/peps/pep-0420/
-.. _Issue #2316: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2316
-.. _PyPA issue: https://github.com/pypa/packaging-problems/issues/12
-.. _develop mode: https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode
-.. _editable install: https://pip.pypa.io/en/stable/reference/pip_install/#editable-installs
-
 *****************************************
 I'm getting weird errors... Can you help?
 *****************************************

MANIFEST.in

@@ -16,7 +16,6 @@
 
 # Generated by synthtool. DO NOT EDIT!
 include README.rst LICENSE
-include google/cloud/spanner_v1/gapic/transports/spanner.grpc.config
 recursive-include google *.json *.proto
 recursive-include tests *
 global-exclude *.py[co]