Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

explicit credentials authentication broken #118

Closed
aydumoulin opened this issue Apr 8, 2021 · 4 comments · Fixed by #120
Closed

explicit credentials authentication broken #118

aydumoulin opened this issue Apr 8, 2021 · 4 comments · Fixed by #120
Assignees
Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.

Comments

@aydumoulin
Copy link

aydumoulin commented Apr 8, 2021

Environment details

  • MacOs 10.15.4
  • Python 3.7.9
  • pybigquery 0.6.0

Steps to reproduce

Create an engine following the example authentication of the readme both for

  • path credentials
  • object credentials

Code example

from sqlalchemy import create_engine
from pybigquery.api import ApiClient

engine = create_engine(uri, credentials_info=secrets)
engine = create_engine(uri, credentials_path=secrets_path)

Stack trace

AttributeError                            Traceback (most recent call last)
<ipython-input-5-90cdb3af0bd1> in <module>
----> 1 engine = create_engine(uri, credentials_info=secrets)

~/.pyenv/versions/3.7.9/lib/python3.7/site-packages/sqlalchemy/engine/__init__.py in create_engine(*args, **kwargs)
    523     strategy = kwargs.pop("strategy", default_strategy)
    524     strategy = strategies.strategies[strategy]
--> 525     return strategy.create(*args, **kwargs)
    526
    527

~/.pyenv/versions/3.7.9/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py in create(self, name_or_url, **kwargs)
     96
     97         # assemble connection arguments
---> 98         (cargs, cparams) = dialect.create_connect_args(u)
     99         cparams.update(pop_kwarg("connect_args", {}))
    100         cargs = list(cargs)  # allow mutability

~/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pybigquery/sqlalchemy_bigquery.py in create_connect_args(self, url)
    365             project_id=project_id,
    366             location=self.location,
--> 367             default_query_job_config=default_query_job_config,
    368         )
    369         return ([client], {})

~/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pybigquery/_helpers.py in create_bigquery_client(credentials_info, credentials_path, default_query_job_config, location, project_id)
     45         )
     46         credentials = credentials.with_scopes(SCOPES)
---> 47         default_project = credentials.project
     48     else:
     49         credentials, default_project = google.auth.default(scopes=SCOPES)

AttributeError: 'Credentials' object has no attribute 'project'

Proposed Explanation

Both approaches above fail at accessing project property of a Credentials instance which does not exist (anymore?)

Proposed Solution

Right now I have monkey patched calls to _helpers.create_bigquery_client this way:

from google.api_core import client_info
import google.auth
from google.cloud import bigquery
from google.oauth2 import service_account

from pybigquery._helpers import USER_AGENT_TEMPLATE, SCOPES, google_client_info
from pybigquery import _helpers


def _patched(
    credentials_info=None,
    credentials_path=None,
    default_query_job_config=None,
    location=None,
    project_id=None,
):
    default_project = None

    if credentials_path:
        credentials = service_account.Credentials.from_service_account_file(
            credentials_path
        )
        credentials = credentials.with_scopes(SCOPES)
        default_project = credentials.project_id
    elif credentials_info:
        credentials = service_account.Credentials.from_service_account_info(
            credentials_info
        )
        credentials = credentials.with_scopes(SCOPES)
        default_project = credentials.project_id
    else:
        credentials, default_project = google.auth.default(scopes=SCOPES)

    if project_id is None:
        project_id = default_project

    return bigquery.Client(
        client_info=google_client_info(),
        project=project_id,
        credentials=credentials,
        location=location,
        default_query_job_config=default_query_job_config,
    )

_helpers.create_bigquery_client = _patched

engine = create_engine(uri, credentials_info=secrets)

The idea is to use credentials.project_id instead of project

Thanks for any remark or help!

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. label Apr 8, 2021
@yoshi-automation yoshi-automation added the triage me I really want to be triaged. label Apr 9, 2021
@tswast tswast added priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. and removed triage me I really want to be triaged. labels Apr 12, 2021
@tswast
Copy link
Collaborator

tswast commented Apr 12, 2021

Thanks for the detailed report and monkeypatch!

It does appear that project_id is the correct property name. https://googleapis.dev/python/google-auth/latest/reference/google.oauth2.service_account.html#google.oauth2.service_account.Credentials.project_id

@tswast
Copy link
Collaborator

tswast commented Apr 12, 2021

PR #120 should fix this. I added some system tests to make sure we're testing with real credentials, not just mocks.

@tswast
Copy link
Collaborator

tswast commented Apr 12, 2021

This fix will be released soon in 0.6.1 #121

@aydumoulin
Copy link
Author

Thanks a lot @tswast ! This was a much appreciated and express fix 😉

It works like a charm I bumped the library to 0.6.1 and removed the patch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/python-bigquery-sqlalchemy API. priority: p1 Important issue which blocks shipping the next release. Will be fixed prior to next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants