Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: address issue in establishing an emulator connection #246

Merged
merged 13 commits into from Mar 19, 2021

Conversation

crwilcox
Copy link
Contributor

The credential and channel establishing for emulators appears to have a defect. Comparing with the recent overhaul in firestore, this should resolve those issues.

Fixes #243 #184

@crwilcox crwilcox requested review from a team as code owners March 15, 2021 21:31
@product-auto-label product-auto-label bot added the api: bigtable Issues related to the googleapis/python-bigtable API. label Mar 15, 2021
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Mar 15, 2021
@crwilcox
Copy link
Contributor Author

TODO: these tests need mock adjustments.
=========================== short test summary info ============================
FAILED tests/unit/test_client.py::Test__create_gapic_client::test_w_emulator
FAILED tests/unit/test_client.py::TestClient::test_constructor_defaults - Ass...
FAILED tests/unit/test_client.py::TestClient::test_constructor_with_emulator_host
3 failed, 1241 passed, 10 warnings in 28.75s

Copy link
Collaborator

@kolea2 kolea2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. Is there a way we can test this as part of the unit tests, by spinning up and emulator instance and running something?

@crwilcox
Copy link
Contributor Author

@kolea2 yep, and likely we should to avoid regressions here.

tests/unit/test_client.py Show resolved Hide resolved
self._emulator_channel = None

if self._emulator_host is not None:
self._emulator_channel = grpc.insecure_channel(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the change moves most of this to a method. Also, using the method derived in firestore, a secure channel can be used.

tests/system.py Outdated
@@ -115,8 +115,12 @@ def setUpModule():
Config.IN_EMULATOR = os.getenv(BIGTABLE_EMULATOR) is not None

if Config.IN_EMULATOR:
credentials = EmulatorCreds()
Config.CLIENT = Client(admin=True, credentials=credentials)
# I expect users won't always pass creds, just use the usual cred flow
Copy link
Contributor Author

@crwilcox crwilcox Mar 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be removed I think. I am not really familiar with why we were doing this, but perhaps it is worth digging into? (this code is 4+ years old)

@crwilcox crwilcox merged commit 1a31826 into master Mar 19, 2021
@crwilcox crwilcox deleted the emulator-channel-setup branch March 19, 2021 20:34
@tseaver
Copy link
Contributor

tseaver commented Sep 27, 2021

@crwilcox I'm not able to run the system_emulated session either locally or on Gihub Workflow.:

$ nox -re system_emulated -- -x
nox > Running session system_emulated
nox > Re-using existing virtual environment at .nox/system_emulated.
Google Cloud SDK 358.0.0
beta 2021.09.17
bigtable 
bq 2.0.71
cloud-datastore-emulator 2.1.0
cloud-firestore-emulator 1.13.0
cloud-spanner-emulator 1.2.0
core 2021.09.17
gsutil 4.68

All components are up to date.
nox > python -m pip install --pre grpcio
Executing: .../google-cloud-sdk/platform/bigtable-emulator/cbtemulator --host=localhost --port=8789
[bigtable] Cloud Bigtable emulator running on 127.0.0.1:8789
nox > python -m pip install mock pytest google-cloud-testutils -c .../src/python-bigtable/testing/constraints-3.8.txt
nox > python -m pip install -e . -c .../python-bigtable/testing/constraints-3.8.txt
nox > py.test --quiet --junitxml=system_3.8_sponge_log.xml tests/system -x
E
==================================== ERRORS ====================================
_____________ ERROR at setup of test_table_read_rows_filter_millis _____________

    @pytest.fixture(scope="session")
    def admin_client():
>       return Client(admin=True)

tests/system/conftest.py:76: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
google/cloud/bigtable/client.py:184: in __init__
    super(Client, self).__init__(
.nox/system_emulated/lib/python3.8/site-packages/google/cloud/client.py:317: in __init__
    Client.__init__(
.nox/system_emulated/lib/python3.8/site-packages/google/cloud/client.py:176: in __init__
    credentials, _ = google.auth.default(scopes=scopes)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

scopes = ('https://www.googleapis.com/auth/bigtable.data', 'https://www.googleapis.com/auth/bigtable.admin')
request = None, quota_project_id = None, default_scopes = None

    def default(scopes=None, request=None, quota_project_id=None, default_scopes=None):
        """Gets the default credentials for the current environment.
    
        `Application Default Credentials`_ provides an easy way to obtain
        credentials to call Google APIs for server-to-server or local applications.
        This function acquires credentials from the environment in the following
        order:
    
        1. If the environment variable ``GOOGLE_APPLICATION_CREDENTIALS`` is set
           to the path of a valid service account JSON private key file, then it is
           loaded and returned. The project ID returned is the project ID defined
           in the service account file if available (some older files do not
           contain project ID information).
    
           If the environment variable is set to the path of a valid external
           account JSON configuration file (workload identity federation), then the
           configuration file is used to determine and retrieve the external
           credentials from the current environment (AWS, Azure, etc).
           These will then be exchanged for Google access tokens via the Google STS
           endpoint.
           The project ID returned in this case is the one corresponding to the
           underlying workload identity pool resource if determinable.
        2. If the `Google Cloud SDK`_ is installed and has application default
           credentials set they are loaded and returned.
    
           To enable application default credentials with the Cloud SDK run::
    
                gcloud auth application-default login
    
           If the Cloud SDK has an active project, the project ID is returned. The
           active project can be set using::
    
                gcloud config set project
    
        3. If the application is running in the `App Engine standard environment`_
           (first generation) then the credentials and project ID from the
           `App Identity Service`_ are used.
        4. If the application is running in `Compute Engine`_ or `Cloud Run`_ or
           the `App Engine flexible environment`_ or the `App Engine standard
           environment`_ (second generation) then the credentials and project ID
           are obtained from the `Metadata Service`_.
        5. If no credentials are found,
           :class:`~google.auth.exceptions.DefaultCredentialsError` will be raised.
    
        .. _Application Default Credentials: https://developers.google.com\
                /identity/protocols/application-default-credentials
        .. _Google Cloud SDK: https://cloud.google.com/sdk
        .. _App Engine standard environment: https://cloud.google.com/appengine
        .. _App Identity Service: https://cloud.google.com/appengine/docs/python\
                /appidentity/
        .. _Compute Engine: https://cloud.google.com/compute
        .. _App Engine flexible environment: https://cloud.google.com\
                /appengine/flexible
        .. _Metadata Service: https://cloud.google.com/compute/docs\
                /storing-retrieving-metadata
        .. _Cloud Run: https://cloud.google.com/run
    
        Example::
    
            import google.auth
    
            credentials, project_id = google.auth.default()
    
        Args:
            scopes (Sequence[str]): The list of scopes for the credentials. If
                specified, the credentials will automatically be scoped if
                necessary.
            request (Optional[google.auth.transport.Request]): An object used to make
                HTTP requests. This is used to either detect whether the application
                is running on Compute Engine or to determine the associated project
                ID for a workload identity pool resource (external account
                credentials). If not specified, then it will either use the standard
                library http client to make requests for Compute Engine credentials
                or a google.auth.transport.requests.Request client for external
                account credentials.
            quota_project_id (Optional[str]): The project ID used for
                quota and billing.
            default_scopes (Optional[Sequence[str]]): Default scopes passed by a
                Google client library. Use 'scopes' for user-defined scopes.
        Returns:
            Tuple[~google.auth.credentials.Credentials, Optional[str]]:
                the current environment's credentials and project ID. Project ID
                may be None, which indicates that the Project ID could not be
                ascertained from the environment.
    
        Raises:
            ~google.auth.exceptions.DefaultCredentialsError:
                If no credentials were found, or if the credentials found were
                invalid.
        """
        from google.auth.credentials import with_scopes_if_required
    
        explicit_project_id = os.environ.get(
            environment_vars.PROJECT, os.environ.get(environment_vars.LEGACY_PROJECT)
        )
    
        checkers = (
            # Avoid passing scopes here to prevent passing scopes to user credentials.
            # with_scopes_if_required() below will ensure scopes/default scopes are
            # safely set on the returned credentials since requires_scopes will
            # guard against setting scopes on user credentials.
            lambda: _get_explicit_environ_credentials(quota_project_id=quota_project_id),
            lambda: _get_gcloud_sdk_credentials(quota_project_id=quota_project_id),
            _get_gae_credentials,
            lambda: _get_gce_credentials(request),
        )
    
        for checker in checkers:
            credentials, project_id = checker()
            if credentials is not None:
                credentials = with_scopes_if_required(
                    credentials, scopes, default_scopes=default_scopes
                )
    
                # For external account credentials, scopes are required to determine
                # the project ID. Try to get the project ID again if not yet
                # determined.
                if not project_id and callable(
                    getattr(credentials, "get_project_id", None)
                ):
                    if request is None:
                        request = google.auth.transport.requests.Request()
                    project_id = credentials.get_project_id(request=request)
    
                if quota_project_id:
                    credentials = credentials.with_quota_project(quota_project_id)
    
                effective_project_id = explicit_project_id or project_id
                if not effective_project_id:
                    _LOGGER.warning(
                        "No project ID could be determined. Consider running "
                        "`gcloud config set project` or setting the %s "
                        "environment variable",
                        environment_vars.PROJECT,
                    )
                return credentials, effective_project_id
    
>       raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
E       google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

.nox/system_emulated/lib/python3.8/site-packages/google/auth/_default.py:488: DefaultCredentialsError
------------------------------ Captured log setup ------------------------------
WARNING  google.auth.compute_engine._metadata:_metadata.py:97 Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: timed out
WARNING  google.auth.compute_engine._metadata:_metadata.py:97 Compute Engine Metadata server unavailable on attempt 2 of 3. Reason: [Errno 113] No route to host
WARNING  google.auth.compute_engine._metadata:_metadata.py:97 Compute Engine Metadata server unavailable on attempt 3 of 3. Reason: timed out
WARNING  google.auth._default:_default.py:286 Authentication failed using Compute Engine authentication due to unavailable metadata server.
- generated xml file: ...python-bigtable/system_3.8_sponge_log.xml -
=========================== short test summary info ============================
ERROR tests/system/test_data_api.py::test_table_read_rows_filter_millis - goo...
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
1 error in 6.17s
nox > Command py.test --quiet --junitxml=system_3.8_sponge_log.xml tests/system -x failed with exit code 1
nox > Session system_emulated failed.

@crwilcox
Copy link
Contributor Author

Feel free to open a new issue. For what it is worth I just pulled main, updated gcloud, and ran this locally. I don't see this error, though I see issues from a pr 2 weeks ago with code to handle emulator v. non-emulator
https://github.com/googleapis/python-bigtable/pull/393/files#diff-6f95a982a6be6964f56636260716af6886d861b8deebc41b1660c41842e9ba5dR159
E UnboundLocalError: local variable 'instance' referenced before assignment

----------------------------------------------- generated xml file: /Users/crwilcox/workspace/python-bigtable/system_3.8_sponge_log.xml ------------------------------------------------
=============================================================================== short test summary info ================================================================================
FAILED tests/system/test_data_api.py::test_access_with_non_admin_client - google.api_core.exceptions.NotFound: 404 table "projects/firestore-harvard-library/instances/g-c-p-d-163277...
FAILED tests/system/test_instance_admin.py::test_instance_exists_hit - google.api_core.exceptions.MethodNotImplemented: 501 unimplemented feature
FAILED tests/system/test_instance_admin.py::test_instance_exists_miss - google.api_core.exceptions.MethodNotImplemented: 501 unimplemented feature
FAILED tests/system/test_instance_admin.py::test_cluster_create - google.api_core.exceptions.MethodNotImplemented: 501 unimplemented feature
ERROR tests/system/test_data_api.py::test_table_read_rows_filter_millis - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_data_api.py::test_table_mutate_rows - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_data_api.py::test_table_truncate - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_data_api.py::test_table_drop_by_prefix - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_data_api.py::test_table_read_rows_w_row_set - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_data_api.py::test_rowset_add_row_range_w_pfx - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_data_api.py::test_table_read_row - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_data_api.py::test_table_read_rows - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_table_admin.py::test_instance_list_tables - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_table_admin.py::test_table_exists - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_table_admin.py::test_table_create - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_table_admin.py::test_table_create_w_families - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_table_admin.py::test_table_create_w_split_keys - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_table_admin.py::test_column_family_create - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_table_admin.py::test_column_family_update - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_table_admin.py::test_column_family_delete - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_table_admin.py::test_table_get_iam_policy - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_table_admin.py::test_table_test_iam_permissions - UnboundLocalError: local variable 'instance' referenced before assignment
ERROR tests/system/test_table_admin.py::test_table_backup - UnboundLocalError: local variable 'instance' referenced before assignment
4 failed, 14 skipped, 19 errors in 3.44s
nox > Command py.test --quiet --junitxml=system_3.8_sponge_log.xml tests/system failed with exit code 1
nox > Session system_emulated failed.

@tseaver
Copy link
Contributor

tseaver commented Sep 27, 2021

@crwilcox Did you run that in a "clean" environment (no GOOGLE_APPLICATION_CREDENTIALS set?)

@crwilcox
Copy link
Contributor Author

I just did now, force unsetting. The code works from a user perspective which is what this PR addressed (Customer CI/CD, not ours)

❯ python
Python 3.9.6 (default, Sep  1 2021, 15:30:45)
[Clang 12.0.5 (clang-1205.0.22.11)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.cloud.bigtable import Client
>>> client = Client(project="notreal", admin=True)
>>> instance = client.instance("instance_id", "location")
>>> instance.list_tables()

[]

@tseaver
Copy link
Contributor

tseaver commented Sep 28, 2021

Hmm, I'm guessing that you have gcloud SDK jumping in here to provide default credentials. I don't keep them configured by default, and they shouldn't be required when running under the emulator:

$ export BIGTABLE_EMULATOR_HOST="localhost:1234"
$ .nox/system-3-8/bin/python
Python 3.8.12 (default, Sep 10 2021, 18:10:10) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.cloud.bigtable import Client
>>> client = Client(project="notreal", admin=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tseaver/projects/agendaless/Google/src/python-bigtable/google/cloud/bigtable/client.py", line 184, in __init__
    super(Client, self).__init__(
  File "/home/tseaver/projects/agendaless/Google/src/python-bigtable/.nox/system-3-8/lib/python3.8/site-packages/google/cloud/client.py", line 317, in __init__
    Client.__init__(
  File "/home/tseaver/projects/agendaless/Google/src/python-bigtable/.nox/system-3-8/lib/python3.8/site-packages/google/cloud/client.py", line 176, in __init__
    credentials, _ = google.auth.default(scopes=scopes)
  File "/home/tseaver/projects/agendaless/Google/src/python-bigtable/.nox/system-3-8/lib/python3.8/site-packages/google/auth/_default.py", line 488, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigtable Issues related to the googleapis/python-bigtable API. cla: yes This human has signed the Contributor License Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bigtable.client shouldn't require credentials when it is connected to the emulator
3 participants