Throw Exception Unknown modifier: $pushAll when run update(push_all)

[dhamada]

This error does not occur when running with docker-compose on macOS, but it occur when running with docker-compose on ubuntu16.04 on windows virtualbox.

How can I avoid this error?

• docker image: python-alpine:3.6, mongo:latest
• docker 17.12
• windows 10
• macOS high sierra

Traceback (most recent call last):
   File "/usr/local/lib/python3.6/site-packages/mongoengine/queryset/base.py", line 514, in update
     upsert=upsert, **write_concern)
   File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 2968, in update
     write_concern, collation=collation)
   File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 835, in _update_retryable
     _update, session)
   File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1099, in _retryable_write
     return self._retry_with_session(retryable, func, s, None)
   File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1076, in _retry_with_session
     return func(session, sock_info, retryable)
   File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 831, in _update
     retryable_write=retryable_write)
   File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 797, in _update
     _check_write_command_response(result)
   File "/usr/local/lib/python3.6/site-packages/pymongo/helpers.py", line 208, in _check_write_command_response
     _raise_last_write_error(write_errors)
   File "/usr/local/lib/python3.6/site-packages/pymongo/helpers.py", line 190, in _raise_last_write_error
     raise WriteError(error.get("errmsg"), error.get("code"), error)
 pymongo.errors.WriteError: Unknown modifier: $pushAll

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "/usr/local/lib/python3.6/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/app/patrobo/pipelines.py", line 88, in close_spider
     self._update_contents_check()
   File "/app/patrobo/pipelines.py", line 142, in _update_contents_check
     self.contents_check.update(push_all__sources=sources)
   File "/usr/local/lib/python3.6/site-packages/mongoengine/document.py", line 571, in update
     return self._qs.filter(**self._object_key).update_one(**kwargs)
   File "/usr/local/lib/python3.6/site-packages/mongoengine/queryset/base.py", line 569, in update_one
     upsert=upsert, multi=False, write_concern=write_concern, **update)
   File "/usr/local/lib/python3.6/site-packages/mongoengine/queryset/base.py", line 525, in update
     raise OperationError('Update failed (%s)' % six.text_type(err))
 mongoengine.errors.OperationError: Update failed (Unknown modifier: $pushAll)

[CalgaryMichael]

I am receiving similar errors when I run tests on master. I was able to find someone running into a similar error on the mongoose repository here.

From this I was led to the official mongodb ticket tracking, where it looks like they have completely removed the $pushAll operator from mongodb v3.6+. Here is the corresponding JIRA ticket.

I was running my tests with an instance of mongodb v3.6.1, which is what I believe is throwing the bug. I am assuming that @dhamada was running on similar version of mongodb. Could you confirm this by running the following in your terminal and posting the output:

mongo --version

This may mean that mongoengine is currently not compatible with the latest version of MongoDB.

A quick look at the PyMongo code shows that the update method that is being called on base.py:513 is deprecated. The solution would be to determine whether to use the methods update_one or update_many depending on whether the many flag has been set.

[CalgaryMichael]

I have begun working on a fix for this issue, but have encountered an issue that I do not believe that I have the authority of correcting.

The new PyMongo methods that are suggested (update_one, update_many) do not handle anything regarding the write concerns. Instead, it looks like the write concerns are being set on instantiation of the Collection object.

A potential way that this may be executed without deprecating the ability to change the write concerns on update would be something like the following:

def update(...):
    ...
    with set_write_concern(queryset._collection, write_concerns):
        result = queryset._collection.update_one(...)
    ...

In this example the set_write_concern would do the following:

• on __enter__: save the existing write concerns, and override it with the values passed in
• on __exit__: restore the previous existing write concerns

This would ensure that when a user overrides the write concerns for a single query, then they do not persist to any subsequent queries.

[CalgaryMichael]

After spending a little more time reviewing the issue, I determined that resolving this would be possible without implementing the calls to PyMongo's update_one and update_many. By changing the way that pushAll is handled in transform.update, I was able to remove the crashing on calling model.objects.update()

The issues that I expressed above should be tackled as a separate issue, as the PyMongo update method that we are calling is deprecated. However, this would be more closely related to #1491

[erdenezul]

related fix is merged