Emulator support broken since 2.1.0 at least on Windows

[lietu]

Environment details

• OS type and version: Windows 10 20H2 build 19042.867
• Python version: 3.9.1
• pip version: 21.0.1
• google-cloud-firestore version: 2.1.0 - 2.1.1 at least are affected

Steps to reproduce

1. pip install google-cloud-firestore in e.g. a virtualenv
2. Run emulator via gcloud beta emulators firestore start --host-port=127.0.0.1:8686
3. Set FIRESTORE_EMULATOR_HOST=127.0.0.1:8686 environment variable
4. Run your code
5. Crash with failed to connect to all addresses on first database access

Crashes near instantly if you use the async client but synchronous client gets it after some lengthy timeout.

If you run this exact same code but pip install google-cloud-firestore==1.9.0, it works perfectly. ==2.0.0 crashes because GOOGLE_APPLICATION_CREDENTIALS is not set? 2.0.1 works, 2.0.2 works.

I tried the myriad of other FIREBASE_FIRESTORE_EMULATOR_ADDRESS etc. and none of them seemed to resolve anything, and the documentation nor the code does not seem to support that this should be necessary. I also tried running the emulator via firebase emulators:start --only firestore but that seemed to be just a downgrade without solving any problems - it didn't allow me to set the port without some configuration file.

The emulator logs show no connection attempts or other issues from the broken client versions.

Code example

Your own example code modified just to work with an emulator in general:

from os import environ
from unittest.mock import Mock

import google.auth.credentials
from google.cloud import firestore


def get_db() -> firestore.Client:
    if environ.get("FIRESTORE_EMULATOR_HOST"):
        print("Connecting to emulator")
        return firestore.Client(
            project="firestore-test",
            credentials=Mock(spec=google.auth.credentials.Credentials),
        )
    else:
        print("Connecting to live environment")
        return firestore.Client()


db = get_db()

# Add a new document
print("Creating document")
doc_ref = db.collection(u'users').document(u'alovelace')
doc_ref.set({  # Crash
    u'first': u'Ada',
    u'last': u'Lovelace',
    u'born': 1815
})

# Then query for documents
print("Reading documents")
users_ref = db.collection(u'users')

for doc in users_ref.stream():
    print(u'{} => {}'.format(doc.id, doc.to_dict()))

Stack trace

From ==2.1.1

Traceback (most recent call last):
  File "C:\source\gcf\venv\lib\site-packages\google\api_core\grpc_helpers.py", line 67, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "C:\source\gcf\venv\lib\site-packages\grpc\_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "C:\source\gcf\venv\lib\site-packages\grpc\_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1621761335.758000000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3009,"referenced_errors":[{"created":"@1621761320.831000000","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\source\gcf\venv\lib\site-packages\google\api_core\retry.py", line 188, in retry_target
    return target()
  File "C:\source\gcf\venv\lib\site-packages\google\api_core\grpc_helpers.py", line 69, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\source\gcf\main.py", line 25, in <module>
    doc_ref.set({
  File "C:\source\gcf\venv\lib\site-packages\google\cloud\firestore_v1\document.py", line 167, in set
    write_results = batch.commit(**kwargs)
  File "C:\source\gcf\venv\lib\site-packages\google\cloud\firestore_v1\batch.py", line 57, in commit
    commit_response = self._client._firestore_api.commit(
  File "C:\source\gcf\venv\lib\site-packages\google\cloud\firestore_v1\services\firestore\client.py", line 836, in commit
    response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
  File "C:\source\gcf\venv\lib\site-packages\google\api_core\gapic_v1\method.py", line 145, in __call__
    return wrapped_func(*args, **kwargs)
  File "C:\source\gcf\venv\lib\site-packages\google\api_core\retry.py", line 285, in retry_wrapped_func
    return retry_target(
  File "C:\source\gcf\venv\lib\site-packages\google\api_core\retry.py", line 203, in retry_target
    six.raise_from(
  File "<string>", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling functools.partial(<function _wrap_unary_errors.<locals>.error_remapped_callable at 0x000001DFD2A20430>, database: "projects/firestore-test/databases/(default)"
writes {
  update {
    name: "projects/firestore-test/databases/(default)/documents/users/alovelace"
    fields {
      key: "born"
      value {
        integer_value: 1815
      }
    }
    fields {
      key: "first"
      value {
        string_value: "Ada"
      }
    }
    fields {
      key: "last"
      value {
        string_value: "Lovelace"
      }
    }
  }
}
, metadata=[('google-cloud-resource-prefix', 'projects/firestore-test/databases/(default)'), ('authorization', 'Bearer owner'), ('x-goog-request-params', 'database=projects/firestore-test/databases/%28default%29'), ('x-goog-api-client', 'gl-python/3.9.1 grpc/1.38.0 gax/1.28.0 gapic/2.1.1')]), last exception: 503 failed to connect to all addresses

[dmahugh]

Thanks for the detailed issue and repro, @lietu
I've confirmed the repro, we'll look into it.

[crwilcox]

Thanks for the report, also, just to simplify the repro, get_db shouldn't be necessary. The Firestore client will, on calls to firestore.Client detect the set FIRESTORE_EMULATOR_HOST and create a channel for this. You shouldn't need to do anything different in your code.

This won't resolve your error, though the following code should result in a repro:

from google.cloud import firestore

db = firestore.Client()

# Add a new document
print("Creating document")
doc_ref = db.collection(u'users').document(u'alovelace')
doc_ref.set({  # Crash
    u'first': u'Ada',
    u'last': u'Lovelace',
    u'born': 1815
})

[lietu]

Maybe that get_db was only necessary pre-2.0.0 or so but it most definitely has been necessary for me in the past. Good if it no longer is the case.

[lietu]

Mjea no it definitely still doesn't work:

Note the crash is about GOOGLE_APPLICATION_CREDENTIALS and triggered on firestore.Client(). This is why the credentials=Mock(spec=google.auth.credentials.Credentials) has been necessary.

[craiglabenz]

I have failed to reproduce this on macOS, and combined with @dmahugh's reproduction on Windows; OS would seem to be a relevant factor.

Continuing to investigate.

[tsikerdekis]

I can verify that this problem does not exist for:
google-cloud-firestore = "==2.0.2"
firebase-admin = "==4.5.0"

[crwilcox]

Assigning dmahugh as I think they are planning to look into this.

[lietu]

I spent a bit of time testing this further and it seems Linux is not affected, quite likely Windows-only.

[lietu]

This is a bit easier to test on for Windows because the async client for some reason crashes immediately instead of waiting for a timeout

import asyncio
from os import environ
from unittest.mock import Mock

import google.auth.credentials
from google.cloud import firestore


def get_db() -> firestore.AsyncClient:
    if environ.get("FIRESTORE_EMULATOR_HOST"):
        print("Connecting to emulator")
        return firestore.AsyncClient(
            project="firestore-test",
            credentials=Mock(spec=google.auth.credentials.Credentials),
        )
    else:
        print("Connecting to live environment")
        return firestore.AsyncClient()


async def main():
    # Add a new document
    print("Creating document")
    doc_ref = db.collection(u'users').document(u'alovelace')
    await doc_ref.set({
        u'first': u'Ada',
        u'last': u'Lovelace',
        u'born': 1815
    })

    # Then query for documents
    print("Reading documents")
    users_ref = db.collection(u'users')

    async for doc in users_ref.stream():
        print(u'{} => {}'.format(doc.id, doc.to_dict()))


if __name__ == "__main__":
    db = get_db()
    asyncio.run(main())

(venv) 18:17:21.89 C:\source\gcf>python asyncmain.py
Connecting to emulator
Creating document
Traceback (most recent call last):
  File "C:\source\gcf\venv\lib\site-packages\google\api_core\grpc_helpers_async.py", line 86, in __await__
    response = yield from self._call.__await__()
  File "C:\source\gcf\venv\lib\site-packages\grpc\aio\_call.py", line 285, in __await__
    raise _create_rpc_error(self._cython_call._initial_metadata,
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1624202248.836000000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3009,"referenced_errors":[{"created":"@1624202248.836000000","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\source\gcf\asyncmain.py", line 41, in <module>
    asyncio.run(main())
  File "c:\python39\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "c:\python39\lib\asyncio\base_events.py", line 642, in run_until_complete
    return future.result()
  File "C:\source\gcf\asyncmain.py", line 25, in main
    await doc_ref.set({
  File "C:\source\gcf\venv\lib\site-packages\google\cloud\firestore_v1\async_document.py", line 131, in set
    write_results = await batch.commit(**kwargs)
  File "C:\source\gcf\venv\lib\site-packages\google\cloud\firestore_v1\async_batch.py", line 58, in commit
    commit_response = await self._client._firestore_api.commit(
  File "C:\source\gcf\venv\lib\site-packages\google\cloud\firestore_v1\services\firestore\async_client.py", line 722, in commit
    response = await rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
  File "C:\source\gcf\venv\lib\site-packages\google\api_core\grpc_helpers_async.py", line 89, in __await__
    raise exceptions.from_grpc_error(rpc_error) from rpc_error
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses

[lietu]

So it seems emulator (or anything?) can't be used with async on Windows since 2.0.0, and because <2.0.0 did not support async that means at all.

1. 2.1.x give the 503 failed to connect to all addresses
2. 2.0.x fail with various "unexpected type of call" errors

The same async example from above crashes with this on 2.0.2

(venv) 12:23:24.97 C:\source\gcf>python asyncmain.py
Connecting to emulator
Creating document
Traceback (most recent call last):
  File "C:\source\gcf\asyncmain.py", line 41, in <module>
    asyncio.run(main())
  File "c:\python39\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "c:\python39\lib\asyncio\base_events.py", line 642, in run_until_complete
    return future.result()
  File "C:\source\gcf\asyncmain.py", line 25, in main
    await doc_ref.set({
  File "C:\source\gcf\venv\lib\site-packages\google\cloud\firestore_v1\async_document.py", line 127, in set
    write_results = await batch.commit(**kwargs)
  File "C:\source\gcf\venv\lib\site-packages\google\cloud\firestore_v1\async_batch.py", line 58, in commit
    commit_response = await self._client._firestore_api.commit(
  File "C:\source\gcf\venv\lib\site-packages\google\cloud\firestore_v1\services\firestore\async_client.py", line 691, in commit
    response = await rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
  File "C:\source\gcf\venv\lib\site-packages\google\api_core\grpc_helpers_async.py", line 180, in error_remapped_callable
    raise TypeError('Unexpected type of call %s' % type(call))
TypeError: Unexpected type of call <class 'google.cloud.firestore_v1.types.firestore.CommitResponse'>

Version 2.0.0 complains about GOOGLE_APPLICATION_CREDENTIALS not being defined, when I create a dummy foo.json file for it and point it there, it gives me the same 503 failed to connect to all addresses error.

[craiglabenz]

Debugged this with @dmahugh for a while and we eventually hit a dead end when the code path turned entirely into compiled C. This is where our adventures took us:

1. Calls from Firestore work their way into grpc and eventually hit this ._blocking() method.
2. That _blocking method prepares the half dozen or so phases of a GRPC call and begins iterating over them with self._channel.segregated_call, which seems to be mostly compiled C code and is completely opaque to pdb.
3. Eventually, the baton is advanced with event = call.next_event(), and printing out state.debug_error_string shows this JSON structure representing the error:

{
  "created":"@1626734356.729000000",
  "description":"Failed to pick subchannel",
  "file":"src/core/ext/filters/client_channel/client_channel.cc",
  "file_line":3009,
  "referenced_errors": [
    {
      "created":"@1626734354.049000000",
      "description":"failed to connect to all addresses",
      "file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc",
      "file_line":398,
      "grpc_status":14
    }
  ]
}

So it seems that something in this compiled C code is not correctly building or reading the server's URL when environment variables have been set to indicate a URL of the form 127.0.0.1:PORT.

[lietu]

So it seems that something in this compiled C code is not correctly building or reading the server's URL when environment variables have been set to indicate a URL of the form 127.0.0.1:PORT.

Sounds like you traced it back to gRPC - I might be naive here, but it sounds to me like it's not gRPC's job to parse FIRESTORE_EMULATOR_HOST environment variables and instead it should be passed to it as an argument from the firestore library, right?

EDIT: Oh hm, I think I misread what you wrote. I also checked with the debugger that in the Python side transport the host was correctly set and the FIRESTORE_EMULATOR_HOST environment variable was properly read, but the abstraction over abstraction over abstraction made me unable to really follow what code gets called when it actually tries to perform the commit so I couldn't see how these things are passed to the C code. I would be a bit surprised if the gRPC library couldn't handle ip:port format for target addresses, but I guess everything is possible.

[craiglabenz]

Those are good points, @lietu. I had wondered if the emulator's networking layer was simply broken on Windows, but I expect that would have surfaced by now in one version of this ticket for each language Firebase supports, so I don't imagine the issue is there.

[lietu]

If this is really an issue with connecting because of secure vs insecure channels in gRPC - I would hope that once this is tracked down that someone goes and creates an issue with gRPC about improving this message to include some information about that, since to me "failed to connect to all addresses" sounds like an issue with parsing the address, whereas something like "failed to connect to all addresses (failed to open secure connection)" or similar would've told me basically instantly that the issue is somewhere with trying to open secure connections to localhost, which is very often the case.

[lietu]

How very strange, it seems something recently changed that fixes this, it seemed to me like it was something in gRPC.

Using https://github.com/lietu/google-cloud-firestore-emulator-bug-test

poetry install
poetry run test.bat

Now both async and sync clients work on Windows as well.

Just prior to pushing that repo I had the same issues with that, but then I ran poetry update and it worked, I managed to record this:

  • Updating google-auth (1.33.0 -> 1.33.1)
  • Updating grpcio (1.38.1 -> 1.39.0)

And if I run poetry add grpcio==1.38.1 and try to run again I get the same error.

Seems like the problem is solved, though it would be nice to know the root cause.

[lietu]

The changelog doesn't seem to hint at the potential solution

The C side included this which is the closest thing I can see that could be related grpc/grpc#26331

[lietu]

Can confirm that a more complex project started working by doing poetry update and getting the same bump for grpcio.

[SulanthaM]

This seem to not have fixed my issue. Here is my env.

Authlib==0.15.4
cachelib==0.2.0
cachetools==4.2.2
certifi==2021.5.30
cffi==1.14.6
charset-normalizer==2.0.3
click==8.0.1
cryptography==3.4.7
Flask==2.0.1
Flask-Session @ git+https://github.com/sulantha2006/flask-session.git@a622380f7c4944e6a79e85060af3193375ca32d5
google-api-core==1.31.0
google-auth==1.33.1
google-cloud-core==1.7.1
google-cloud-firestore==2.1.3
google-cloud-secret-manager==2.6.0
googleapis-common-protos==1.53.0
grpc-google-iam-v1==0.12.3
grpcio==1.39.0
gunicorn==20.1.0
idna==3.2
itsdangerous==2.0.1
Jinja2==3.0.1
libcst==0.3.19
MarkupSafe==2.0.1
mock==4.0.3
mypy-extensions==0.4.3
packaging==21.0
proto-plus==1.19.0
protobuf==3.17.3
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pyparsing==2.4.7
python-dotenv==0.18.0
pytz==2021.1
PyYAML==5.4.1
requests==2.26.0
rsa==4.7.2
six==1.16.0
typing-extensions==3.10.0.0
typing-inspect==0.7.1
urllib3==1.26.6
Werkzeug==2.0.1

[lietu]

This seem to not have fixed my issue. Here is my env.

Strange, also I couldn't reproduce any issue on Mac or Linux when tested, so your issue might be different from mine.

Have you checked that

1. the emulator is actually running and listening to connections and not e.g. blocked by a firewall or similar
2. FIRESTORE_EMULATOR_HOST is pointing to the right ip & port

?

[SulanthaM]

More detail:
I am running the app through docker-compose like:

version: "3.7"
services:
  firestore:
    image: sulantha/firestore-emulator:latest
    environment:
      - FIRESTORE_PROJECT_ID=dummy-project
      - PORT=8200
    ports:
      - '8200:8200'
  app:
    environment:
      - FIRESTORE_EMULATOR_HOST=firestore:8200
      - FIRESTORE_PROJECT_ID=dummy-project
      - ENVIRONMENT=DEVELOPMENT
      - PORT=8080
      - ENV=DEV
    build:
      context: .
    volumes:
      - type: bind
        source: ~/Downloads/ebi-ai-visread-e6da7b3f7a90.json
        target: /tmp/keys/auth.json
    ports:
      - '8080:8080'
    depends_on:
      - firestore

And connecting to Firestore like:

if env == 'DEV':
    print("Using Dev Envs ###############")
    FS_PROJECT = "dummy-project"
    os.environ["FIRESTORE_DATASET"] = "sessions"
    os.environ["FIRESTORE_EMULATOR_HOST"] = "firestore:8200"
    os.environ["FIRESTORE_EMULATOR_HOST_PATH"] = "firestore:8200/firestore"
    os.environ["FIRESTORE_HOST"] = "http://firestore:8200"
    os.environ["FIRESTORE_PROJECT_ID"] = "dummy-project"
    fs_client = firestore.Client(project=FS_PROJECT, credentials=mock.Mock(spec=google.auth.credentials.Credentials))
    sm_client = secretmanager.SecretManagerServiceClient.from_service_account_json('/tmp/keys/auth.json')

I am also connecting to secret manager with a correct credential file, but I don't think that could be an issue.

App Start:

app_1        | [2021-07-22 15:14:52 +0000] [1] [INFO] Starting gunicorn 20.1.0
firestore_1  | Updated property [core/project].
firestore_1  | Executing: /google-cloud-sdk/platform/cloud-firestore-emulator/cloud_firestore_emulator start --host=0.0.0.0 --port=8200
firestore_1  | [firestore] API endpoint: http://0.0.0.0:8200
app_1        | [2021-07-22 15:14:52 +0000] [1] [INFO] Listening at: http://0.0.0.0:8080 (1)
app_1        | [2021-07-22 15:14:52 +0000] [1] [INFO] Using worker: gthread
firestore_1  | [firestore] If you are using a library that supports the FIRESTORE_EMULATOR_HOST environment variable, run:
app_1        | [2021-07-22 15:14:52 +0000] [7] [INFO] Booting worker with pid: 7
firestore_1  | [firestore] 
firestore_1  | [firestore]    export FIRESTORE_EMULATOR_HOST=0.0.0.0:8200
firestore_1  | [firestore] 
firestore_1  | [firestore] Dev App Server is now running.
firestore_1  | [firestore] 
app_1        | Using Dev Envs ###############

[SulanthaM]

I created a testing project for ease of test. Would it be possible to try to see where the issue here is.

https://github.com/sulantha2006/FS_EMU_Test

This seem to not have fixed my issue. Here is my env.

Strange, also I couldn't reproduce any issue on Mac or Linux when tested, so your issue might be different from mine.

Have you checked that

1. the emulator is actually running and listening to connections and not e.g. blocked by a firewall or similar
2. FIRESTORE_EMULATOR_HOST is pointing to the right ip & port

?

I created a testing project for ease of test. Would it be possible to try to see where the issue here is.

https://github.com/sulantha2006/FS_EMU_Test

[lietu]

@SulanthaM can you confirm if docker build . works on https://github.com/lietu/google-cloud-firestore-emulator-bug-test for you?

It installs the emulator on the same container as it runs the tests on.

If so, this could be e.g. the secure/insecure channel selection not working when not using localhost/127.0.0.1 or some other issue in your setup. The error messages given by gRPC are kind of .. trash .. at least in this instance.

[crwilcox]

Fix merged in #402. Will be present in the next release (already present if installed from master)

[lietu]

Fix merged in #402. Will be present in the next release (already present if installed from master)

That seems to have been unrelated, as this was fixed without using master and only updating grpcio.

#359 (comment)

[SulanthaM]

Fix merged in #402. Will be present in the next release (already present if installed from master)

Using master fixed the issue for me but, now I see this error in mocking credentials.

app_1        | Traceback (most recent call last):
app_1        |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2070, in wsgi_app
app_1        |     response = self.full_dispatch_request()
app_1        |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1515, in full_dispatch_request
app_1        |     rv = self.handle_user_exception(e)
app_1        |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1513, in full_dispatch_request
app_1        |     rv = self.dispatch_request()
app_1        |   File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1499, in dispatch_request
app_1        |     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
app_1        |   File "/app/main.py", line 16, in index
app_1        |     doc_ref.set(data)
app_1        |   File "/usr/local/lib/python3.9/site-packages/google/cloud/firestore_v1/document.py", line 167, in set
app_1        |     write_results = batch.commit(**kwargs)
app_1        |   File "/usr/local/lib/python3.9/site-packages/google/cloud/firestore_v1/batch.py", line 57, in commit
app_1        |     commit_response = self._client._firestore_api.commit(
app_1        |   File "/usr/local/lib/python3.9/site-packages/google/cloud/firestore_v1/client.py", line 104, in _firestore_api
app_1        |     return self._firestore_api_helper(
app_1        |   File "/usr/local/lib/python3.9/site-packages/google/cloud/firestore_v1/base_client.py", line 151, in _firestore_api_helper
app_1        |     channel = self._emulator_channel(transport)
app_1        |   File "/usr/local/lib/python3.9/site-packages/google/cloud/firestore_v1/base_client.py", line 182, in _emulator_channel
app_1        |     if self._credentials is not None and self._credentials.id_token is not None:
app_1        |   File "/usr/local/lib/python3.9/site-packages/mock/mock.py", line 632, in __getattr__
app_1        |     raise AttributeError("Mock object has no attribute %r" % name)
app_1        | AttributeError: Mock object has no attribute 'id_token'

This is my code.

dev_creds = mock.Mock(spec=Credentials)
db = firestore.Client(project="dummy-project", credentials=dev_creds)

[lietu]

Mjea, this is now broken again but in a very different way.

poetry add git+https://github.com/googleapis/python-firestore google-cloud-firestore

Breaks both async and sync client, with that above issue.

The workaround is:

def get_db() -> firestore.AsyncClient:
    if environ.get("FIRESTORE_EMULATOR_HOST"):
        print("Connecting to emulator")
        credentials = Mock(spec=google.auth.credentials.Credentials)
        credentials.id_token = ""
        return firestore.AsyncClient(
            project="firestore-test",
            credentials=credentials,
        )

It would also be nice if the earlier claim was true about not needing to override the credentials for emulator, but it's not: #359 (comment)

poetry add google-cloud-firestore==2.1.3  # or 2.1.0, 2.1.1, or 2.1.2

Fixes the problem reported in this issue with async client connecting to the local emulator on Windows, as that was an issue caused by grpcio which has been updated to fix the issue. >=2.0.0,<2.1.0 still doesn't work because of other bugs in there.

[lietu]

This issue seems resolved to me, it was somehow resolved in grpcio, and whatever patch is involved with the id_token thing is a separate issue.

It would be very nice still to guarantee that in the future we get better and more descriptive errors than "failed to connect to all addresses".

[SulanthaM]

@lietu Thanks, Your workaround to set id_token in credentials fixed the issue and I will use it until that's officially fixed.

[lietu]

I assume the maintainers of this project know better where to raise the open issue of getting reasonable error messages instead of "failed to connect to all addresses", such as

• "Failed to connect to all addresses, connection timed out"
• "Failed to connect to all addresses, SSL handshake failed"
• "Failed to connect to all addresses, connection refused"
• etc.

[crwilcox]

It wasn't unrelated and I validated your issue and resolved it via this. It is possible grpc fixed the fact that local channel credentials + secure channel didn't work on windows though, which is what my change routed around ;)

It may no longer be necessary, but it is definitely related. :)

[crwilcox]

@SulanthaM the issue you likely ran into is the emulator wasn't providing the header for token, which would leave off an authorization header, and then be rejected as an invalid request. My fix to move to insecure also included this change.

[crwilcox]

To validate: @lietu is indicating the following combintation works on windows:

google-cloud-firestore==2.1.3
grpcio==1.39.0

We should validate this.

it sounds like even with this, id_token needs to be set in credentials which is a change we could make internally with emulator credentials.

@dmahugh could you validate this on your windows machine?

(while we did resolve this reopening just in case we really can revert)

[dmahugh]

I've tried upgrading grpcio to 1.39.0, and it still failed on my Windows 10 machine, but then installing #402 makes it work on my machine. @lietu could you post a pip freeze of your entire environment? Perhaps there's another difference that is affecting things.

[crwilcox]

Thanks @dmahugh for looking into this. For now I am going to assume the fix works, though I am curious to find out what fixed it without for @lietu

Thanks!

[lietu]

https://github.com/lietu/google-cloud-firestore-emulator-bug-test

with start_emulator.bat running in one terminal (just sets it to a predictable localhost:8686)

virtualenv venv
pip install google-cloud-firestore==2.1.0 grpcio==1.39.0
set FIRESTORE_EMULATOR_HOST=localhost:8686
python asyncmain.py

works

pip install grpcio==1.38.1
python asyncmain.py

fails

pip freeze for some reason lists quite a few things installed outside the virtualenv but I guess this is a start anyway

while working:

appdirs==1.4.4
black==21.5b1
CacheControl==0.12.6
cachetools==4.2.1
certifi==2020.12.5
cffi==1.14.6
cfgv==3.2.0
chardet==4.0.0
charset-normalizer==2.0.3
click==7.1.2
codespell==2.1.0
colorama==0.4.4
distlib==0.3.1
filelock==3.0.12
gcrc==1.0.1
google-api-core==1.26.1
google-auth==1.27.1
google-cloud-core==1.6.0
google-cloud-firestore==2.1.0
googleapis-common-protos==1.53.0
grpcio==1.39.0
identify==1.5.13
idna==2.10
iso8601==0.1.14
msgpack==1.0.2
mypy-extensions==0.4.3
nodeenv==1.5.0
packaging==20.9
pathspec==0.8.1
pip-licenses==3.3.1
pre-commit==2.10.1
proto-plus==1.17.0
protobuf==3.15.6
PTable==0.9.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pydantic==1.8.2
pyparsing==2.4.7
python-dateutil==2.8.2
python-dotenv==0.18.0
pytz==2021.1
PyYAML==5.4.1
regex==2021.4.4
requests==2.26.0
rsa==4.7.2
-e git+https://github.com/lietu/shylock.git@658e935cf95c3be0c87e1178db1648f5d55e3e66#egg=shylock
six==1.15.0
toml==0.10.2
typer==0.3.2
typing-extensions==3.10.0.0
urllib3==1.26.3
virtualenv==20.4.2

while failing:

appdirs==1.4.4
black==21.5b1
CacheControl==0.12.6
cachetools==4.2.1
certifi==2020.12.5
cffi==1.14.6
cfgv==3.2.0
chardet==4.0.0
charset-normalizer==2.0.3
click==7.1.2
codespell==2.1.0
colorama==0.4.4
distlib==0.3.1
filelock==3.0.12
gcrc==1.0.1
google-api-core==1.26.1
google-auth==1.27.1
google-cloud-core==1.6.0
google-cloud-firestore==2.1.0
googleapis-common-protos==1.53.0
grpcio==1.38.1
identify==1.5.13
idna==2.10
iso8601==0.1.14
msgpack==1.0.2
mypy-extensions==0.4.3
nodeenv==1.5.0
packaging==20.9
pathspec==0.8.1
pip-licenses==3.3.1
pre-commit==2.10.1
proto-plus==1.17.0
protobuf==3.15.6
PTable==0.9.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pydantic==1.8.2
pyparsing==2.4.7
python-dateutil==2.8.2
python-dotenv==0.18.0
pytz==2021.1
PyYAML==5.4.1
regex==2021.4.4
requests==2.26.0
rsa==4.7.2
-e git+https://github.com/lietu/shylock.git@658e935cf95c3be0c87e1178db1648f5d55e3e66#egg=shylock
six==1.15.0
toml==0.10.2
typer==0.3.2
typing-extensions==3.10.0.0
urllib3==1.26.3
virtualenv==20.4.2

--- success     2021-07-26 23:44:57.908742000 +0300
+++ fail        2021-07-26 23:44:50.190927400 +0300
@@ -18,7 +18,7 @@
 google-cloud-core==1.6.0
 google-cloud-firestore==2.1.0
 googleapis-common-protos==1.53.0
-grpcio==1.39.0
+grpcio==1.38.1
 identify==1.5.13
 idna==2.10
 iso8601==0.1.14

not sure how you're testing but this is how I create my async client https://github.com/lietu/google-cloud-firestore-emulator-bug-test/blob/991829f3897aa59da1e290f0f275a6f3218b7c6a/asyncmain.py#L9-L17