chore: update templates (#97)

* chore(python): skip docfx in main presubmit * fix: properly template the repo name Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Fri Jan 8 10:32:13 2021 -0700 Source-Repo: googleapis/synthtool Source-Sha: fb53b6fb373b7c3edf4e55f3e8036bc6d73fa483 Source-Link: googleapis/synthtool@fb53b6f * chore: add missing quotation mark Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Mon Jan 11 09:43:06 2021 -0700 Source-Repo: googleapis/synthtool Source-Sha: 16ec872dd898d7de6e1822badfac32484b5d9031 Source-Link: googleapis/synthtool@16ec872 * chore: add 3.9 to noxfile template Since the python-docs-samples noxfile-template doesn't sync with this, I wanted to make sure the noxfile template matched the most recent change [here](https://github.com/GoogleCloudPlatform/python-docs-samples/pull/4968/files) cc @tmatsuo Source-Author: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Source-Date: Fri Jan 15 17:24:05 2021 -0800 Source-Repo: googleapis/synthtool Source-Sha: 56ddc68f36b32341e9f22c2c59b4ce6aa3ba635f Source-Link: googleapis/synthtool@56ddc68 * build(python): make `NOX_SESSION` optional I added this accidentally in #889. `NOX_SESSION` should be passed down if it is set but not marked required. Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Tue Jan 19 09:38:04 2021 -0700 Source-Repo: googleapis/synthtool Source-Sha: ba960d730416fe05c50547e975ce79fcee52c671 Source-Link: googleapis/synthtool@ba960d7 * chore: Add header checker config to python library synth Now that we have it working in [python-docs-samples](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/.github/header-checker-lint.yml) we should consider adding it to the 🐍 libraries :) Source-Author: Leah E. Cole <6719667+leahecole@users.noreply.github.com> Source-Date: Mon Jan 25 13:24:08 2021 -0800 Source-Repo: googleapis/synthtool Source-Sha: 573f7655311b553a937f9123bee17bf78497db95 Source-Link: googleapis/synthtool@573f765 * chore: add noxfile parameters for extra dependencies Also, add tests for some noxfile parameters for assurance that the template generates valid Python. Co-authored-by: Jeffrey Rennie <rennie@google.com> Source-Author: Tim Swast <swast@google.com> Source-Date: Tue Jan 26 12:26:57 2021 -0600 Source-Repo: googleapis/synthtool Source-Sha: 778d8beae28d6d87eb01fdc839a4b4d966ed2ebe Source-Link: googleapis/synthtool@778d8be * build: migrate to flakybot Source-Author: Justin Beckwith <justin.beckwith@gmail.com> Source-Date: Thu Jan 28 22:22:38 2021 -0800 Source-Repo: googleapis/synthtool Source-Sha: d1bb9173100f62c0cfc8f3138b62241e7f47ca6a Source-Link: googleapis/synthtool@d1bb917 * chore(python): include py.typed files in release A py.typed file must be included in the released package for it to be considered typed by type checkers. https://www.python.org/dev/peps/pep-0561/#packaging-type-information. See googleapis/python-secret-manager#79 Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Fri Feb 5 17:32:06 2021 -0700 Source-Repo: googleapis/synthtool Source-Sha: 33366574ffb9e11737b3547eb6f020ecae0536e8 Source-Link: googleapis/synthtool@3336657 * docs: update python contributing guide Adds details about blacken, updates version for system tests, and shows how to pass through pytest arguments. Source-Author: Chris Cotter <cjcotter@google.com> Source-Date: Mon Feb 8 17:13:36 2021 -0500 Source-Repo: googleapis/synthtool Source-Sha: 4679e7e415221f03ff2a71e3ffad75b9ec41d87e Source-Link: googleapis/synthtool@4679e7e * build(python): enable flakybot on library unit and system tests Source-Author: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com> Source-Date: Wed Feb 17 14:10:46 2021 -0700 Source-Repo: googleapis/synthtool Source-Sha: d17674372e27fb8f23013935e794aa37502071aa Source-Link: googleapis/synthtool@d176743 * test: install pyopenssl for mtls testing Source-Author: arithmetic1728 <58957152+arithmetic1728@users.noreply.github.com> Source-Date: Tue Mar 2 12:27:56 2021 -0800 Source-Repo: googleapis/synthtool Source-Sha: 0780323da96d5a53925fe0547757181fe76e8f1e Source-Link: googleapis/synthtool@0780323
googleapis · Mar 23, 2021 · 9fd02a6 · 9fd02a6
1 parent b78db75
commit 9fd02a6
Show file tree

Hide file tree

Showing 12 changed files with 110 additions and 24 deletions.
diff --git a/.github/header-checker-lint.yml b/.github/header-checker-lint.yml
@@ -0,0 +1,15 @@
+{"allowedCopyrightHolders": ["Google LLC"],
+ "allowedLicenses": ["Apache-2.0", "MIT", "BSD-3"],
+ "ignoreFiles": ["**/requirements.txt", "**/requirements-test.txt"],
+ "sourceFileExtensions": [
+ 	"ts", 
+ 	"js", 
+ 	"java", 
+ 	"sh", 
+ 	"Dockerfile", 
+ 	"yaml", 
+ 	"py",
+ 	"html",
+ 	"txt"
+ ]
+}
diff --git a/.gitignore b/.gitignore
@@ -50,8 +50,10 @@ docs.metadata
 
 # Virtual environment
 env/
+
+# Test logs
 coverage.xml
-sponge_log.xml
+*sponge_log.xml
 
 # System test environment variables.
 system_tests/local_test_setup

diff --git a/.kokoro/build.sh b/.kokoro/build.sh
@@ -15,7 +15,11 @@
 
 set -eo pipefail
 
-cd github/python-documentai
+if [[ -z "${PROJECT_ROOT:-}" ]]; then
+    PROJECT_ROOT="github/python-documentai"
+fi
+
+cd "${PROJECT_ROOT}"
 
 # Disable buffering, so that the logs stream through.
 export PYTHONUNBUFFERED=1
@@ -30,16 +34,26 @@ export GOOGLE_APPLICATION_CREDENTIALS=${KOKORO_GFILE_DIR}/service-account.json
 export PROJECT_ID=$(cat "${KOKORO_GFILE_DIR}/project-id.json")
 
 # Remove old nox
-python3.6 -m pip uninstall --yes --quiet nox-automation
+python3 -m pip uninstall --yes --quiet nox-automation
 
 # Install nox
-python3.6 -m pip install --upgrade --quiet nox
-python3.6 -m nox --version
+python3 -m pip install --upgrade --quiet nox
+python3 -m nox --version
+
+# If this is a continuous build, send the test log to the FlakyBot.
+# See https://github.com/googleapis/repo-automation-bots/tree/master/packages/flakybot.
+if [[ $KOKORO_BUILD_ARTIFACTS_SUBDIR = *"continuous"* ]]; then
+  cleanup() {
+    chmod +x $KOKORO_GFILE_DIR/linux_amd64/flakybot
+    $KOKORO_GFILE_DIR/linux_amd64/flakybot
+  }
+  trap cleanup EXIT HUP
+fi
 
 # If NOX_SESSION is set, it only runs the specified session,
 # otherwise run all the sessions.
 if [[ -n "${NOX_SESSION:-}" ]]; then
-    python3.6 -m nox -s "${NOX_SESSION:-}"
+    python3 -m nox -s ${NOX_SESSION:-}
 else
-    python3.6 -m nox
+    python3 -m nox
 fi
diff --git a/.kokoro/docs/docs-presubmit.cfg b/.kokoro/docs/docs-presubmit.cfg
@@ -15,3 +15,14 @@ env_vars: {
     key: "TRAMPOLINE_IMAGE_UPLOAD"
     value: "false"
 }
+
+env_vars: {
+    key: "TRAMPOLINE_BUILD_FILE"
+    value: "github/python-documentai/.kokoro/build.sh"
+}
+
+# Only run this nox session.
+env_vars: {
+    key: "NOX_SESSION"
+    value: "docs docfx"
+}
diff --git a/.kokoro/test-samples.sh b/.kokoro/test-samples.sh
@@ -87,11 +87,11 @@ for file in samples/**/requirements.txt; do
     python3.6 -m nox -s "$RUN_TESTS_SESSION"
     EXIT=$?
 
-    # If this is a periodic build, send the test log to the Build Cop Bot.
-    # See https://github.com/googleapis/repo-automation-bots/tree/master/packages/buildcop.
+    # If this is a periodic build, send the test log to the FlakyBot.
+    # See https://github.com/googleapis/repo-automation-bots/tree/master/packages/flakybot.
     if [[ $KOKORO_BUILD_ARTIFACTS_SUBDIR = *"periodic"* ]]; then
-      chmod +x $KOKORO_GFILE_DIR/linux_amd64/buildcop
-      $KOKORO_GFILE_DIR/linux_amd64/buildcop
+      chmod +x $KOKORO_GFILE_DIR/linux_amd64/flakybot
+      $KOKORO_GFILE_DIR/linux_amd64/flakybot
     fi
 
     if [[ $EXIT -ne 0 ]]; then

diff --git a/.kokoro/trampoline_v2.sh b/.kokoro/trampoline_v2.sh
@@ -159,7 +159,7 @@ if [[ -n "${KOKORO_BUILD_ID:-}" ]]; then
 	"KOKORO_GITHUB_COMMIT"
 	"KOKORO_GITHUB_PULL_REQUEST_NUMBER"
 	"KOKORO_GITHUB_PULL_REQUEST_COMMIT"
-	# For Build Cop Bot
+	# For FlakyBot
 	"KOKORO_GITHUB_COMMIT_URL"
 	"KOKORO_GITHUB_PULL_REQUEST_URL"
     )

diff --git a/.trampolinerc b/.trampolinerc
@@ -24,6 +24,7 @@ required_envvars+=(
 pass_down_envvars+=(
     "STAGING_BUCKET"
     "V2_STAGING_BUCKET"
+    "NOX_SESSION"
 )
 
 # Prevent unintentional override on the default image.

diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst
@@ -70,9 +70,14 @@ We use `nox <https://nox.readthedocs.io/en/latest/>`__ to instrument our tests.
 - To test your changes, run unit tests with ``nox``::
 
     $ nox -s unit-2.7
-    $ nox -s unit-3.7
+    $ nox -s unit-3.8
     $ ...
 
+- Args to pytest can be passed through the nox command separated by a `--`. For
+  example, to run a single test::
+
+    $ nox -s unit-3.8 -- -k <name of test>
+
   .. note::
 
     The unit tests and system tests are described in the
@@ -93,8 +98,12 @@ On Debian/Ubuntu::
 ************
 Coding Style
 ************
+- We use the automatic code formatter ``black``. You can run it using
+  the nox session ``blacken``. This will eliminate many lint errors. Run via::
+
+   $ nox -s blacken
 
-- PEP8 compliance, with exceptions defined in the linter configuration.
+- PEP8 compliance is required, with exceptions defined in the linter configuration.
   If you have ``nox`` installed, you can test that you have not introduced
   any non-compliant code via::
 
@@ -133,13 +142,18 @@ Running System Tests
 
 - To run system tests, you can execute::
 
-   $ nox -s system-3.7
+   # Run all system tests
+   $ nox -s system-3.8
    $ nox -s system-2.7
 
+   # Run a single system test
+   $ nox -s system-3.8 -- -k <name of test>
+
+
   .. note::
 
       System tests are only configured to run under Python 2.7 and
-      Python 3.7. For expediency, we do not run them in older versions
+      Python 3.8. For expediency, we do not run them in older versions
       of Python 3.
 
   This alone will not run the tests. You'll need to change some local

diff --git a/MANIFEST.in b/MANIFEST.in
@@ -16,10 +16,10 @@
 
 # Generated by synthtool. DO NOT EDIT!
 include README.rst LICENSE
-recursive-include google *.json *.proto
+recursive-include google *.json *.proto py.typed
 recursive-include tests *
 global-exclude *.py[co]
 global-exclude __pycache__
 
 # Exclude scripts for samples readmegen
-prune scripts/readme-gen
+prune scripts/readme-gen
diff --git a/noxfile.py b/noxfile.py
@@ -30,6 +30,17 @@
 SYSTEM_TEST_PYTHON_VERSIONS = ["3.8"]
 UNIT_TEST_PYTHON_VERSIONS = ["3.6", "3.7", "3.8", "3.9"]
 
+# 'docfx' is excluded since it only needs to run in 'docs-presubmit'
+nox.options.sessions = [
+    "unit",
+    "system",
+    "cover",
+    "lint",
+    "lint_setup_py",
+    "blacken",
+    "docs",
+]
+
 
 @nox.session(python=DEFAULT_PYTHON_VERSION)
 def lint(session):
@@ -75,12 +86,14 @@ def default(session):
     session.install(
         "mock", "pytest", "pytest-cov",
     )
+
     session.install("-e", ".")
 
     # Run py.test against the unit tests.
     session.run(
         "py.test",
         "--quiet",
+        f"--junitxml=unit_{session.python}_sponge_log.xml",
         "--cov=google/cloud",
         "--cov=tests/unit",
         "--cov-append",
@@ -110,6 +123,9 @@ def system(session):
     # Sanity check: Only run tests if the environment variable is set.
     if not os.environ.get("GOOGLE_APPLICATION_CREDENTIALS", ""):
         session.skip("Credentials must be set via environment variable")
+    # Install pyopenssl for mTLS testing.
+    if os.environ.get("GOOGLE_API_USE_CLIENT_CERTIFICATE", "false") == "true":
+        session.install("pyopenssl")
 
     system_test_exists = os.path.exists(system_test_path)
     system_test_folder_exists = os.path.exists(system_test_folder_path)
@@ -129,9 +145,21 @@ def system(session):
 
     # Run py.test against the system tests.
     if system_test_exists:
-        session.run("py.test", "--quiet", system_test_path, *session.posargs)
+        session.run(
+            "py.test",
+            "--quiet",
+            f"--junitxml=system_{session.python}_sponge_log.xml",
+            system_test_path,
+            *session.posargs,
+        )
     if system_test_folder_exists:
-        session.run("py.test", "--quiet", system_test_folder_path, *session.posargs)
+        session.run(
+            "py.test",
+            "--quiet",
+            f"--junitxml=system_{session.python}_sponge_log.xml",
+            system_test_folder_path,
+            *session.posargs,
+        )
 
 
 @nox.session(python=DEFAULT_PYTHON_VERSION)

diff --git a/samples/snippets/noxfile.py b/samples/snippets/noxfile.py
@@ -85,7 +85,7 @@ def get_pytest_env_vars() -> Dict[str, str]:
 
 # DO NOT EDIT - automatically generated.
 # All versions used to tested samples.
-ALL_VERSIONS = ["2.7", "3.6", "3.7", "3.8"]
+ALL_VERSIONS = ["2.7", "3.6", "3.7", "3.8", "3.9"]
 
 # Any default versions that should be ignored.
 IGNORED_VERSIONS = TEST_CONFIG['ignored_versions']

diff --git a/synth.metadata b/synth.metadata
@@ -4,7 +4,7 @@
       "git": {
         "name": ".",
         "remote": "https://github.com/googleapis/python-documentai.git",
-        "sha": "d6f183a696b211c6d29bc28e9bbd0a8537f65577"
+        "sha": "b78db75d0f3c68a38fbc5e540d264f11f5abfd3b"
       }
     },
     {
@@ -19,14 +19,14 @@
       "git": {
         "name": "synthtool",
         "remote": "https://github.com/googleapis/synthtool.git",
-        "sha": "41a4e56982620d3edcf110d76f4fcdfdec471ac8"
+        "sha": "0780323da96d5a53925fe0547757181fe76e8f1e"
       }
     },
     {
       "git": {
         "name": "synthtool",
         "remote": "https://github.com/googleapis/synthtool.git",
-        "sha": "41a4e56982620d3edcf110d76f4fcdfdec471ac8"
+        "sha": "0780323da96d5a53925fe0547757181fe76e8f1e"
       }
     }
   ],
@@ -58,6 +58,7 @@
     ".github/ISSUE_TEMPLATE/feature_request.md",
     ".github/ISSUE_TEMPLATE/support_request.md",
     ".github/PULL_REQUEST_TEMPLATE.md",
+    ".github/header-checker-lint.yml",
     ".github/release-please.yml",
     ".github/snippet-bot.yml",
     ".gitignore",