pymongo.errors.InvalidOperation: Cannot use MongoClient after close

[letoss]

Hello, community.

I recently upgraded the version of MongoEngine and Pymongo and I started having an issue while running my tests. I'm getting the error described in the title:
pymongo.errors.InvalidOperation: Cannot use MongoClient after close

I have this issue while running my tests with pytest. I will try to explain in detail all my findings to see if someone here can tell me if it's MongoEngine related or not (I think it is).

First, I upgraded Pymongo to 4.3.3 and MongoEngine to 0.27.1. Then I tried downgrading to 4.0.0 and 0.24. Still the same issue.

This is the list of tools I'm using:

• FastAPI
• Pymongo
• MongoEngine
• Pytest

This is the following structure I have:

app.py

from mongoengine.connection import connect, disconnect

@app.on_event("startup")
async def startup():
    app.db = connect(host=settings.get_mongo_dsn(), alias="default")

@app.on_event("shutdown")
def shutdown_event():
    disconnect()

conftest.py

from .models import create_default_data


@pytest.fixture
def test_app(request):
    with TestClient(app) as test_app:
        print(f"APP DB list: {app.db.list_database_names()}")
        # create_default_data uses MongoEngine over a collection to create the instances.
        create_default_data()
        yield test_app

        app.db.drop_database(settings.MONGO_DB)

The folder structure of my project is like this:
src/
--- spa/
--- api/

So, my findings... I started debugging mongoEngine and also Pymongo. I discovered that Pymongo has this Topology object. I started monitoring the ID from the current instance since this is the one that manages the connection and ends up raising that error I'm getting.

If I run pytest from src/api/, the first test runs successfully, but the following ones start failing. I checked and it seems to be that the ID from the instance of the Topology is always pointing to the one from the test that was run before it (which has been closed by the disconnect).
If I run pytest from src, this doesn't happen, but it will not take the pytest.ini settings and then run over the local DB instead the testing one.

Also, the outdated Topology ID is only happening inside create_default_data. If you check the print above, that will also use Topology and it will have the current ID. So my hunch is that, the collections defined at models.py keep the same instance from the previous tests and then the Topology reference is to the old one.

I hope anything of what I wrote makes sense to you since I'm running out of ideas what is going on.

BTW, this is working correctly in versions:

• MongoEngine 0.23.1
• Pymongo 3.12.3

Edit:

Debugging I found that it looks like the queryset is the one keeping the old reference:

ipdb> MyCollection.objects._collection_obj._Collection__database.client.topology_description
<TopologyDescription id: 64916fde42e944527572eac7, topology_type: Single, servers: [<ServerDescription ('mongo', 27017) server_type: Unknown, rtt: None>]>
ipdb> conn = _connections["default"]
ipdb> conn.get_default_database().client.topology_description
<TopologyDescription id: 649170ce42e944527572eacc, topology_type: Single, servers: [<ServerDescription ('mongo', 27017) server_type: Standalone, rtt: 0.000535320856554299>]>

Edit 2:

I found out that the topology ID is changing between tests. But the one that tries to really connect is the first one created on the first test.
I went back to MongoEngine 0.23.1 and it looks like it's doing the same, but the difference is that the first topology connection is never really closed, so it can be still used.

I'm not sure what changed between 0.23.1 and 0.27.0, but it looks like now those connections are really closed and they can't be used. Is there any suggestion?

Thanks