New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: address issue in establishing an emulator connection #246
Conversation
TODO: these tests need mock adjustments. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good. Is there a way we can test this as part of the unit tests, by spinning up and emulator instance and running something?
@kolea2 yep, and likely we should to avoid regressions here. |
self._emulator_channel = None | ||
|
||
if self._emulator_host is not None: | ||
self._emulator_channel = grpc.insecure_channel( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the change moves most of this to a method. Also, using the method derived in firestore, a secure channel can be used.
89e93f5
to
ced9620
Compare
tests/system.py
Outdated
@@ -115,8 +115,12 @@ def setUpModule(): | |||
Config.IN_EMULATOR = os.getenv(BIGTABLE_EMULATOR) is not None | |||
|
|||
if Config.IN_EMULATOR: | |||
credentials = EmulatorCreds() | |||
Config.CLIENT = Client(admin=True, credentials=credentials) | |||
# I expect users won't always pass creds, just use the usual cred flow |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This could be removed I think. I am not really familiar with why we were doing this, but perhaps it is worth digging into? (this code is 4+ years old)
@crwilcox I'm not able to run the $ nox -re system_emulated -- -x
nox > Running session system_emulated
nox > Re-using existing virtual environment at .nox/system_emulated.
Google Cloud SDK 358.0.0
beta 2021.09.17
bigtable
bq 2.0.71
cloud-datastore-emulator 2.1.0
cloud-firestore-emulator 1.13.0
cloud-spanner-emulator 1.2.0
core 2021.09.17
gsutil 4.68
All components are up to date.
nox > python -m pip install --pre grpcio
Executing: .../google-cloud-sdk/platform/bigtable-emulator/cbtemulator --host=localhost --port=8789
[bigtable] Cloud Bigtable emulator running on 127.0.0.1:8789
nox > python -m pip install mock pytest google-cloud-testutils -c .../src/python-bigtable/testing/constraints-3.8.txt
nox > python -m pip install -e . -c .../python-bigtable/testing/constraints-3.8.txt
nox > py.test --quiet --junitxml=system_3.8_sponge_log.xml tests/system -x
E
==================================== ERRORS ====================================
_____________ ERROR at setup of test_table_read_rows_filter_millis _____________
@pytest.fixture(scope="session")
def admin_client():
> return Client(admin=True)
tests/system/conftest.py:76:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
google/cloud/bigtable/client.py:184: in __init__
super(Client, self).__init__(
.nox/system_emulated/lib/python3.8/site-packages/google/cloud/client.py:317: in __init__
Client.__init__(
.nox/system_emulated/lib/python3.8/site-packages/google/cloud/client.py:176: in __init__
credentials, _ = google.auth.default(scopes=scopes)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
scopes = ('https://www.googleapis.com/auth/bigtable.data', 'https://www.googleapis.com/auth/bigtable.admin')
request = None, quota_project_id = None, default_scopes = None
def default(scopes=None, request=None, quota_project_id=None, default_scopes=None):
"""Gets the default credentials for the current environment.
`Application Default Credentials`_ provides an easy way to obtain
credentials to call Google APIs for server-to-server or local applications.
This function acquires credentials from the environment in the following
order:
1. If the environment variable ``GOOGLE_APPLICATION_CREDENTIALS`` is set
to the path of a valid service account JSON private key file, then it is
loaded and returned. The project ID returned is the project ID defined
in the service account file if available (some older files do not
contain project ID information).
If the environment variable is set to the path of a valid external
account JSON configuration file (workload identity federation), then the
configuration file is used to determine and retrieve the external
credentials from the current environment (AWS, Azure, etc).
These will then be exchanged for Google access tokens via the Google STS
endpoint.
The project ID returned in this case is the one corresponding to the
underlying workload identity pool resource if determinable.
2. If the `Google Cloud SDK`_ is installed and has application default
credentials set they are loaded and returned.
To enable application default credentials with the Cloud SDK run::
gcloud auth application-default login
If the Cloud SDK has an active project, the project ID is returned. The
active project can be set using::
gcloud config set project
3. If the application is running in the `App Engine standard environment`_
(first generation) then the credentials and project ID from the
`App Identity Service`_ are used.
4. If the application is running in `Compute Engine`_ or `Cloud Run`_ or
the `App Engine flexible environment`_ or the `App Engine standard
environment`_ (second generation) then the credentials and project ID
are obtained from the `Metadata Service`_.
5. If no credentials are found,
:class:`~google.auth.exceptions.DefaultCredentialsError` will be raised.
.. _Application Default Credentials: https://developers.google.com\
/identity/protocols/application-default-credentials
.. _Google Cloud SDK: https://cloud.google.com/sdk
.. _App Engine standard environment: https://cloud.google.com/appengine
.. _App Identity Service: https://cloud.google.com/appengine/docs/python\
/appidentity/
.. _Compute Engine: https://cloud.google.com/compute
.. _App Engine flexible environment: https://cloud.google.com\
/appengine/flexible
.. _Metadata Service: https://cloud.google.com/compute/docs\
/storing-retrieving-metadata
.. _Cloud Run: https://cloud.google.com/run
Example::
import google.auth
credentials, project_id = google.auth.default()
Args:
scopes (Sequence[str]): The list of scopes for the credentials. If
specified, the credentials will automatically be scoped if
necessary.
request (Optional[google.auth.transport.Request]): An object used to make
HTTP requests. This is used to either detect whether the application
is running on Compute Engine or to determine the associated project
ID for a workload identity pool resource (external account
credentials). If not specified, then it will either use the standard
library http client to make requests for Compute Engine credentials
or a google.auth.transport.requests.Request client for external
account credentials.
quota_project_id (Optional[str]): The project ID used for
quota and billing.
default_scopes (Optional[Sequence[str]]): Default scopes passed by a
Google client library. Use 'scopes' for user-defined scopes.
Returns:
Tuple[~google.auth.credentials.Credentials, Optional[str]]:
the current environment's credentials and project ID. Project ID
may be None, which indicates that the Project ID could not be
ascertained from the environment.
Raises:
~google.auth.exceptions.DefaultCredentialsError:
If no credentials were found, or if the credentials found were
invalid.
"""
from google.auth.credentials import with_scopes_if_required
explicit_project_id = os.environ.get(
environment_vars.PROJECT, os.environ.get(environment_vars.LEGACY_PROJECT)
)
checkers = (
# Avoid passing scopes here to prevent passing scopes to user credentials.
# with_scopes_if_required() below will ensure scopes/default scopes are
# safely set on the returned credentials since requires_scopes will
# guard against setting scopes on user credentials.
lambda: _get_explicit_environ_credentials(quota_project_id=quota_project_id),
lambda: _get_gcloud_sdk_credentials(quota_project_id=quota_project_id),
_get_gae_credentials,
lambda: _get_gce_credentials(request),
)
for checker in checkers:
credentials, project_id = checker()
if credentials is not None:
credentials = with_scopes_if_required(
credentials, scopes, default_scopes=default_scopes
)
# For external account credentials, scopes are required to determine
# the project ID. Try to get the project ID again if not yet
# determined.
if not project_id and callable(
getattr(credentials, "get_project_id", None)
):
if request is None:
request = google.auth.transport.requests.Request()
project_id = credentials.get_project_id(request=request)
if quota_project_id:
credentials = credentials.with_quota_project(quota_project_id)
effective_project_id = explicit_project_id or project_id
if not effective_project_id:
_LOGGER.warning(
"No project ID could be determined. Consider running "
"`gcloud config set project` or setting the %s "
"environment variable",
environment_vars.PROJECT,
)
return credentials, effective_project_id
> raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
E google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
.nox/system_emulated/lib/python3.8/site-packages/google/auth/_default.py:488: DefaultCredentialsError
------------------------------ Captured log setup ------------------------------
WARNING google.auth.compute_engine._metadata:_metadata.py:97 Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
WARNING google.auth.compute_engine._metadata:_metadata.py:97 Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: [Errno 113] No route to host
WARNING google.auth.compute_engine._metadata:_metadata.py:97 Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
WARNING google.auth._default:_default.py:286 Authentication failed using Compute Engine authentication due to unavailable metadata server.
- generated xml file: ...python-bigtable/system_3.8_sponge_log.xml -
=========================== short test summary info ============================
ERROR tests/system/test_data_api.py::test_table_read_rows_filter_millis - goo...
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
1 error in 6.17s
nox > Command py.test --quiet --junitxml=system_3.8_sponge_log.xml tests/system -x failed with exit code 1
nox > Session system_emulated failed. |
Feel free to open a new issue. For what it is worth I just pulled main, updated gcloud, and ran this locally. I don't see this error, though I see issues from a pr 2 weeks ago with code to handle emulator v. non-emulator
|
@crwilcox Did you run that in a "clean" environment (no |
I just did now, force unsetting. The code works from a user perspective which is what this PR addressed (Customer CI/CD, not ours)
|
Hmm, I'm guessing that you have
|
The credential and channel establishing for emulators appears to have a defect. Comparing with the recent overhaul in firestore, this should resolve those issues.
Fixes #243 #184