Firestore error when paginating with document snapshot with 'null' field

[guanzo]

Environment details

• OS type and version: Windows 10
• Python version: 3.6.8
• google-cloud-firestore version: 2.1.3

Steps to reproduce

1. Try to paginate a query with a document snapshot and where clause.

Code example

def getPayments ():
    col = firestoreClient.collection('payments')
    query = col.where('paymentId', '==', None).limit(10).order_by('__name__')

    lastDoc = None
    for doc in query.stream():
        lastDoc = doc

    query = col.where('paymentId', '==', None).limit(10).order_by('__name__').start_after(lastDoc)

    lastDoc = None
    for doc in query.stream(): # Errors here
        lastDoc = doc

Stack trace

Traceback (most recent call last):
  File "\project-et4z2F\lib\site-packages\google\api_core\grpc_helpers.py", line 160, in error_remapped_callable
    return _StreamingResponseIterator(result, prefetch_first_result=prefetch_first)
  File "\project-et4z2F\lib\site-packages\google\api_core\grpc_helpers.py", line 83, in __init__      
    self._stored_first_result = six.next(self._wrapped)
  File "\project-et4z2F\lib\site-packages\grpc\_channel.py", line 426, in __next__
    return self._next()
  File "\project-et4z2F\lib\site-packages\grpc\_channel.py", line 826, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.INVALID_ARGUMENT
        details = "order by clause cannot contain more fields after the key"
        debug_error_string = "{"created":"@1629750504.795000000","description":"Error received from peer ipv6:[2607:f8b0:4005:805::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1067,"grpc_message":"order by clause cannot contain more fields after the 
key","grpc_status":3}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "./transactions/pay_publishers.py", line 359, in <module>
    main()
  File "./transactions/pay_publishers.py", line 353, in main
    payoutMultiplier=payoutMultiplier
  File "./transactions/pay_publishers.py", line 115, in payPublishers
    pendingPayouts = firestore_db.getPayments()
  File "C:\Users\Eric\workspace\arc-core\common\db\firestore_db.py", line 211, in getPendingPayouts
    for doc in query.stream():
  File "\project-et4z2F\lib\site-packages\google\cloud\firestore_v1\query.py", line 209, in stream    
    request=request, metadata=self._client._rpc_metadata, **kwargs,
  File "\project-et4z2F\lib\site-packages\google\cloud\firestore_v1\services\firestore\client.py", line 945, in run_query
    response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
  File "\project-et4z2F\lib\site-packages\google\api_core\gapic_v1\method.py", line 145, in __call__  
    return wrapped_func(*args, **kwargs)
  File "\project-et4z2F\lib\site-packages\google\api_core\retry.py", line 290, in retry_wrapped_func  
    on_error=on_error,
  File "\project-et4z2F\lib\site-packages\google\api_core\retry.py", line 188, in retry_target        
    return target()
  File "\project-et4z2F\lib\site-packages\google\api_core\grpc_helpers.py", line 162, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 order by clause cannot contain more fields after the key

On the 2nd query.stream(), i get this error: google.api_core.exceptions.InvalidArgument: 400 order by clause cannot contain more fields after the key.

I followed the pagination documentation, but added a where clause which broke it. If I remove the where clauses it works fine.

[guanzo]

..After much trial and error, I got it working by changing .start_after(lastDoc) to start_after({ '__name__': lastDoc.id })

Not sure why that works but it does, the error went away.

[crwilcox]

I did some investigation and it seems particular to using the null type field.

Using empty strings and not null value works, so there may be something particular about ingesting when a field is a null type.

Also, stumbled on an interesting corner case where lastDoc was null that we can fix. Will open a separate bug for that. (#439)

Here is a standalone Repro though for this bug that should reproduce:

from google.cloud import firestore
import datetime

client = firestore.Client()
col = client.collection('0-test-collection-0')
col.add({"paymentId": None})

query = col.where('paymentId', '==', None).limit(10).order_by('__name__')

lastDoc = None
for doc in query.stream():
    print(doc)
    lastDoc = doc

query = col.where('paymentId', '==', None).limit(10).order_by('__name__').start_after(lastDoc)

lastDoc = None
for doc in query.stream(): # Errors here
    lastDoc = doc

[tseaver]

@crwilcox Any relationship here to your fix in PR #440?

[crwilcox]

#440 fixed #439 which was opened by me when I saw that issue related to this one. That part is resolved. The issue with null fields isn't though.