Datastore: Client requires authentication even when connecting to the datastore emulator #11

rylz · 2018-08-03T00:25:36Z

When I try to construct a datastore client on a box that has the datastore emulator enabled and running, I get the following:

  File ".../model/datastore.py", line 6, in <module>
    client = datastore.Client()
  File ".../lib/python3.6/site-packages/google/cloud/datastore/client.py", line 205, in __init__
    project=project, credentials=credentials, _http=_http)
  File ".../lib/python3.6/site-packages/google/cloud/client.py", line 215, in __init__
    _ClientProjectMixin.__init__(self, project=project)
  File ".../lib/python3.6/site-packages/google/cloud/client.py", line 169, in __init__
    project = self._determine_default(project)
  File ".../lib/python3.6/site-packages/google/cloud/datastore/client.py", line 222, in _determine_default
    return _determine_default_project(project)
  File ".../lib/python3.6/site-packages/google/cloud/datastore/client.py", line 75, in _determine_default_project
    project = _base_default_project(project=project)
  File ".../lib/python3.6/site-packages/google/cloud/_helpers.py", line 179, in _determine_default_project
    _, project = google.auth.default()
  File ".../lib/python3.6/site-packages/google/auth/_default.py", line 283, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or
explicitly create credential and re-run the application. For more

However, it's clear that these credentials are not actually required because when I initialize my client as follows, it works in both dev and production:

if os.environ.get(environment_vars.GCD_HOST):
    # avoid a bug in the google client - it tries to authenticate even when the emulator is enabled
    client = datastore.Client(_http=requests.Session)
else:
    client = datastore.Client()

With that change, the library works fine for me, but this seems like something the library ought to be able to detect and handle by default to save devs a lot of time.

In case they're relevant:

OS type and version

$ uname -a
Darwin Rileys-MacBook-Pro.local 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64

Python version and virtual environment information python --version

$ python --version
Python 3.6.5

google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze

$ pip show google-cloud-datastore
Name: google-cloud-datastore
Version: 1.7.0

The text was updated successfully, but these errors were encountered:

tseaver · 2018-08-03T14:05:24Z

@rylz Thanks for the report! See also #4840 .

cmin764 · 2019-02-28T16:23:12Z

The problem is still open, because with some code like this:

client_kwargs = {}
if settings.DEBUG:
    import requests
    client_kwargs.update({
        "project": settings.PROJECT_ID,
        "_http": requests.Session(),
    })
client = datastore.Client(**client_kwargs)

(note that I had to pass the project too just to make sure it is safely inferred)
I'm still getting:

google.api_core.exceptions.ServiceUnavailable: 503 Getting metadata from plugin failed with error: 'NoneType' object has no attribute 'before_request'

This is happening because somehow the resulted credentials (which is None) are used under the gRPC transport (or other) where .before_request is called. I've also tried to pass some mock EmulatorCreds as credentials, but still, it tries to make the authentication and if we solve the issue on the initial requestor side, it comes back on the other steps.

Is there any way of running Datastore emulator within dev_appserver.py using standard python3.7 runtime and being able to store an entity? I'm using a similar environment as described above.

Thanks!

sumit-ql · 2019-08-09T17:44:56Z

@rylz Are you still facing the issue with latest release.

I tried to connect to datastore emulator using AnonymousCredentials and its working for me.

Here is the steps i followed and sample code:

Start datastore emulator using gcloud sdk.
Set the env variable : DATASTORE_EMULATOR_HOST
create datastore client with AnonymousCredentials
After that i am able to put one Entity into datastore.

OS : Win7 x64
Python : 3.7.3
google-cloud-datastore: 1.9.0

Sample Code

from google.cloud import datastore
from google.auth.credentials import AnonymousCredentials

client = datastore.Client(
    project="emulated-project",
    namespace='ns_test',
    credentials=AnonymousCredentials(),
    _http=None
)

print ("Adding a entity in datastore....")
pdf = datastore.Entity(client.key('PDF'))
pdf.update({'path': 'asd'})
client.put(pdf)
print ("Success..")

zmcddn · 2019-08-09T19:04:10Z

@sumit-ql I figured it out pretty much the same time as you did. I think in addition to your sample code, there are two required arguments in the Client() function, which is project and credentials. So I was able to get it working by client = datastore.Client(project="emulated-project", credentials=AnonymousCredentials(),)

HemangChothani · 2020-02-12T12:52:53Z

Closing this issue, because it has been resolved.
It's working fine without passing credentials=AnonymousCredentials() as parameter.

feel free to reopen if needed!

c4rlo · 2020-08-10T20:51:25Z

I'm experiencing this issue with google-cloud-datastore 1.14.0.

It only works with credentials=AnonymousCredentials().

In addition to that I found I needed to export GOOGLE_CLOUD_PROJECT; passing project= did not work.

c4rlo · 2020-08-10T20:51:57Z

feel free to reopen if needed!

I would, but lack privileges to do so.

HemangChothani · 2020-08-11T13:31:07Z

@c4rlo Cloud you please share google-cloud-sdk version, i am not able to reproduce it.

Here are the steps:

Start datastore emulator using gcloud sdk.
Set the env variable : DATASTORE_EMULATOR_HOST
import datastore and create client

from google.cloud import datastore
from google.auth.credentials import AnonymousCredentials

client = datastore.Client(
    project="emulated-project",
    namespace='ns_test',
    credentials=AnonymousCredentials(),
    _http=None
)

tseaver · 2020-08-11T18:38:43Z

In addition to passing an instance of AnonymousCredentials(), you could also use the EmulatorCreds class from google-test-utils, e.g.:

/path/to/venv/bin/pip install google-test-utils

from google.cloud import datastore
from test_utils.system import EmulatorCreds

client = datastore.Client(
    project="emulated-project",
    namespace='ns_test',
    credentials=EmulatorCreds(),
    _http=None
)

c4rlo · 2020-08-12T07:59:56Z

I use google-cloud-sdk version 1.14.0, currently the latest.

As I mentioned, when explicitly passing credentials=AnonymousCredentials(), it works fine (disregarding the project= issue for now).

However, I don't see a requirement for that documented anywhere. It's not mentioned either in the emulator docs or in the google-cloud-datastore docs, as far as I can tell. How are users supposed to find out about this?

Moreover, why should this even be necessary? As the original comment in this issue said:

this seems like something the library ought to be able to detect and handle by default to save devs a lot of time

HemangChothani · 2020-08-13T05:21:02Z

@c4rlo For emulator you can use export DATASTORE_PROJECT_ID=my-project-id to declare the project or you can pass the project as a parameter.

c4rlo · 2020-08-13T07:54:27Z

Ok, export DATASTORE_PROJECT_ID=my-project-id does in fact work correctly for me now. Thanks!

However, that was not really the main issue.

The main issue is the need to pass credentials=AnonymousCredentials() (or I suppose EmulatorCreds() as mentioned). Why does the library not infer this automatically given I have DATASTORE_EMULATOR_HOST set? (Along with four other DATASTORE_* environment variables set by $(gcloud beta emulators datastore env-init).)

And if there is a good reason for this, then why is the need for this not documented anywhere?

HemangChothani · 2020-08-13T09:29:11Z

The main issue is the need to pass credentials=AnonymousCredentials() (or I suppose EmulatorCreds() as mentioned). Why does the library not infer this automatically given I have DATASTORE_EMULATOR_HOST set?

@c4rlo I think it's new feature request to set at EmulatorCreds automatically if found DATASTO RE_EMULATOR_HOST is set.

@tseaver Your view on this, python-spanner has done the same thing like the user demands.

c4rlo · 2020-08-15T16:09:30Z

I see this is now fixed in 1.15.0 via #71. Many thanks, @tseaver!

crwilcox transferred this issue from googleapis/google-cloud-python Feb 7, 2020

crwilcox added api: datastore Issues related to the googleapis/python-datastore API. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. labels Feb 7, 2020

HemangChothani closed this as completed Feb 12, 2020

HemangChothani reopened this Aug 11, 2020

tseaver closed this as completed Aug 11, 2020

tseaver mentioned this issue Aug 13, 2020

Normalize emulator support #70

Closed

3 tasks

This was referenced Dec 5, 2020

bigtable emulator and google-cloud-happybase googleapis/google-cloud-python#10515

Closed

bigtable.client for CI/CD emulator without credentials googleapis/python-bigtable#184

Closed

JustinBeckwith assigned HemangChothani Feb 1, 2021

steve-kidreads-app mentioned this issue Feb 25, 2021

Throwing exception in CI/CD build environment because no default credentials googleapis/python-secret-manager#82

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Datastore: Client requires authentication even when connecting to the datastore emulator #11

Datastore: Client requires authentication even when connecting to the datastore emulator #11

rylz commented Aug 3, 2018

tseaver commented Aug 3, 2018

cmin764 commented Feb 28, 2019 •

edited

sumit-ql commented Aug 9, 2019 •

edited

zmcddn commented Aug 9, 2019 •

edited

HemangChothani commented Feb 12, 2020

c4rlo commented Aug 10, 2020

c4rlo commented Aug 10, 2020

HemangChothani commented Aug 11, 2020

tseaver commented Aug 11, 2020

c4rlo commented Aug 12, 2020

HemangChothani commented Aug 13, 2020 •

edited

c4rlo commented Aug 13, 2020

HemangChothani commented Aug 13, 2020

c4rlo commented Aug 15, 2020

Datastore: Client requires authentication even when connecting to the datastore emulator #11

Datastore: Client requires authentication even when connecting to the datastore emulator #11

Comments

rylz commented Aug 3, 2018

tseaver commented Aug 3, 2018

cmin764 commented Feb 28, 2019 • edited

sumit-ql commented Aug 9, 2019 • edited

zmcddn commented Aug 9, 2019 • edited

HemangChothani commented Feb 12, 2020

c4rlo commented Aug 10, 2020

c4rlo commented Aug 10, 2020

HemangChothani commented Aug 11, 2020

tseaver commented Aug 11, 2020

c4rlo commented Aug 12, 2020

HemangChothani commented Aug 13, 2020 • edited

c4rlo commented Aug 13, 2020

HemangChothani commented Aug 13, 2020

c4rlo commented Aug 15, 2020

cmin764 commented Feb 28, 2019 •

edited

sumit-ql commented Aug 9, 2019 •

edited

zmcddn commented Aug 9, 2019 •

edited

HemangChothani commented Aug 13, 2020 •

edited