Unable to create Firestore client when user is authorized

[mik-laj]

I cannot create a client for firebase when using an authorized HTTP client. The problem does not occur when I am anonymous.

Environment details

• OS: Mac OS
• Python version: Python 3.6.8 (v3.6.8:3c6b436a57, Dec 24 2018, 02:04:31) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
• pip version: pip 19.0.3
• google-api-python-client version: google-api-python-client==1.7.10

Steps to reproduce

When I ran following code

import google_auth_httplib2

from googleapiclient import discovery
import httplib2

import google.oauth2.service_account

key_path='files/gcp/keys/sa.json'

credentials = google.oauth2.service_account.Credentials.from_service_account_file(key_path, scopes=[
    "https://www.googleapis.com/auth/cloud-platform",
])

http = httplib2.Http()
authed_http = google_auth_httplib2.AuthorizedHttp(credentials, http=http)

print("Non authorized client: ")
try:
    print(discovery.build("firestore", "v1"))
except:
    print("Unable to create client")

print("Authorized client: ")
try:
    print(discovery.build("firestore", "v1", http=authed_http))
except Exception as e:

    print("Unable to create client")

I got folllowing message;

Non authorized client:
<googleapiclient.discovery.Resource object at 0x10c8dc390>
Authorized client:
Unable to create client
<HttpError 400 when requesting https://www.googleapis.com/discovery/v1/apis/firestore/v1/rest returned "Request contains an invalid argument.">

I am currently using the following hack, which allows me to build a client that works properly and uses authorization.

print("Hack:")
try:
    non_authorized_conn = discovery.build("firestore", "v1", cache_discovery=False)
    client = discovery.build_from_document(
        non_authorized_conn._rootDesc,  # pylint: disable=protected-access
        http=authed_http
    )
    print(client)
except Exception as e:
    print("Unable to create client")

[orar7598]

Hi mik :)
Had this issue as well. I think this module doesn't build the credentials the way google-api-python-client requesting it.
Try the following:

# Import discovery from 'google-api-client'
from googleapiclient import discovery
# Import GoogleCredentials Class oath2client.client
from oauth2client.client import GoogleCredentials
# Path to JSON File
jsonPath = "jsonCredentialsFile.json"
# Building credentials from stream
credentials = GoogleCredentials.from_stream(jsonPath)
# Define the service
try:
    service = discovery.build('firestore', 'v1', credentials=credentials)
    print(service)
# Throw exception if authentication failed
except Exception as err:
    raise Exception("Could not authenticate. See details: {}".format(err))

[mik-laj]

But why do all other services work properly?

[mik-laj]

The oauth2client package is deprecated. I can't use it in my project.

[busunkim96]

Hi @mik-laj,

I was able to reproduce the error, but I'll have to do some more investigation on why it is happening.

If possible, please use the modern firestore client google-cloud-firestore (https://github.com/googleapis/python-firestore).

You can pass the google-auth credentials directly to discovery.build. Could you try this?

credentials = google.oauth2.service_account.Credentials.from_service_account_file(key_path, scopes=[
    "https://www.googleapis.com/auth/compute",
])

print(discovery.build("firestore", "v1", credentials=credentials))

On an unrelated note you will probably want to adjust the scopes. Requests to the FIrestore REST API need the https://www.googleapis.com/auth/datastore scope. documentation

[mik-laj]

@busunkim96 I cannot pass credentials directly to the library because I want to set up a user agent.

I know there is another library for Firebase. Here is my PR. which adds this information to README.md.
#833

According to this documentation, I use the correct scope.

Authorization Scopes
Requires one of the following OAuth scopes:

https://www.googleapis.com/auth/datastore
https://www.googleapis.com/auth/cloud-platform

https://firebase.google.com/docs/firestore/reference/rest/v1beta1/projects.databases/exportDocuments

I discovered this error while working on the integration of Airflow and Firestore.
My PR: apache/airflow#7725
Code responsible for creating the API client:
https://github.com/apache/airflow/blob/63a3102/airflow/providers/google/firebase/hooks/firestore.py#L68-L79
Code responsible for creating the HTTP client:
https://github.com/apache/airflow/blob/63a3102ede8fb8f764d251b20cad5ee5bef84f50/airflow/providers/google/cloud/hooks/base.py#L218-L227
Code responsible for creating credentials;
https://github.com/apache/airflow/blob/63a3102ede8fb8f764d251b20cad5ee5bef84f50/airflow/providers/google/cloud/hooks/base.py#L145-L203

[busunkim96]

I saw that you had the compute scope in your original post and didn't correct it when I replied. Apologies if that caused confusion. 😄

Are you trying to use Firebase or Firestore? There is a firestore specific library: https://github.com/googleapis/python-firestore. The one you linked in the pull request is https://github.com/firebase/firebase-admin-python.

If user-agent is the only blocker you can pass that via client_info in the modern clients. See
https://googleapis.dev/python/firestore/latest/client.html

I'll go do some more investigation on AuthorizedHttp.

[mik-laj]

I want to use Firestore, more precisely, I need the export_documents method. This method is not available in the google-cloud library according to documentation
but I looked at the source code and found that this method is available.
https://github.com/googleapis/python-firestore/blob/master/google/cloud/firestore_admin_v1/gapic/firestore_admin_client.py#L648-L656
This will probably solve my problem completely, but now we have another problem - holes in documentation.

[mik-laj]

Here is ticket about lack of documentation for FirestoreAdminClient; googleapis/python-firestore#30

[busunkim96]

I think this is a discovery bug. Googlers, see internal issue 170905603.

Summary:

The firestore discovery docs cannot be retrieved when there the GET request has extra headers. This occurs for all the versions of firestore (v1, v1beta1, v1beta2) using both the v1 and v2 style discovery URIs.

$ curl -H "authorization: Bearer foo"  https://firestore.googleapis.com/\$discovery/rest?version=v1
{
  "error": {
    "code": 400,
    "message": "Request contains an invalid argument.",
    "status": "INVALID_ARGUMENT"
  }
}
$ curl -H "authorization: Bearer foo"  https://www.googleapis.com/discovery/v1/apis/firestore/v1/rest
{
  "error": {
    "code": 400,
    "message": "Request contains an invalid argument.",
    "status": "INVALID_ARGUMENT"
  }
}

This does not seem to cause an issue for other APIs.

$ curl -H "authorization: Bearer foo"  https://datastore.googleapis.com/\$discovery/rest?version=v1
...
$ curl -H "authorization: Bearer foo"  https://www.googleapis.com/discovery/v1/apis/datastore/v1/rest
...

[parthea]

Hi @mik-laj ,
We've just released version 2.0 of google-api-python-client. One of the key features is that discovery.build() will retrieve discovery documents from the client library itself rather than fetching them over the internet. Please try version 2.0 and let us know if you're still experiencing an issue. I'm going to close off this issue due to inactivity. If you're still having trouble, please re-open it with more information.