New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fetching a document takes very long when it has a large map with a large index #351
Comments
@Cvratingen Can you show the full output of |
I updated the protobuf package.
I've since changed the structure of my setup but created a simple test document with the following map
Fetching this document took me 392.7523 sec |
I can confirm what seems an abnormally long time to fetch the document, after only a relatively short interval (a few seconds?) to save it: >>> import time
>>> from google.cloud.firestore_v1 import Client
>>> client = Client()
>>> collection = client.collection("repro-351")
>>> document = collection.document("doc-id")
>>> data = {"a": {f"{x}": x for x in range(19200)}}
>>> document.set(data)
>>> before = time.time(); snapshot = document.get(); after = time.time()
>>> print(f"Time: {after - before} seconds")
Time: 193.5851743221283 seconds
>>> snapshot.to_dict() == data
True |
Hmm, |
Running the same >>> import time
>>> from google.cloud.firestore_v1 import Client
>>> client = Client()
>>> collection = client.collection("repro-351")
>>> data = {"a": {f"{x}": x for x in range(19200)}}
>>> document = collection.document("doc-id")
>>> before = time.time(); snapshot = document.get(); after = time.time()
>>> print(f"Time: {after - before} seconds")
Time: 0.7956831455230713 seconds
>>> snapshot.to_dict() == data
True |
* chore: remove obsolete skip for old Python 3 versions * perf: strip proto wrappers in '_helpers.decode_{value,dict}' Closes #351.
We have a database where we store information for some IOT devices.
Now our devices routinely send out a value every [5 or 60 or 300] seconds. I'm using a firestore (native mode) database to save these values.
I've got a program which aggregates the messages and updates all the documents every ~10 minutes with all messages received.
The structure of the database is as following:
The document "readings_10-05-2021" will have a map with the values
I'm trying to retreive this document with:
When the map has only 1 value it takes about ~.19 seconds
When the map has 1440 values it takes about ~1.5 seconds
When the map has 19200 values it takes about ~400 seconds
I've tested this with documents of the roughly same size in bytes to account for download speeds.
note: fetching this document with the large map on the google cloud terminal takes ~1 second.
As advised by @samtstern posting this here.
Environment details
Python 3.9.2
pip 21.1.1
google-cloud-firestore 2.1.1
The text was updated successfully, but these errors were encountered: