Automatically recreate ChangeStream on 'close' event

[vkarpov15]

Discussed in #13602

^{Originally posted by thaoula July 11, 2023}
Hi All,

We are trying to utilise ChangeStreams using Mongoose and have encountered some challenges with reliability.

We have the following issues:

• For some unknown reason, our DB level ChangeStream which is connected to an Atlas Cluster stops receiving changes. We do not get any errors even though we are using ChangeStream.on('error') listener. Nothing is logged.

• When this happens, restarting the container but keeping the resume token i.e we are setting resumeAfter does not work. We only get changes after deleting the resume token and restarting the container.

Before logging some Github issues I wanted to see if anyone has experienced problems like this and whether you have some tips.

What Works

1. We are able to successfully open a ChangeStream at the database level using connection.watch().
2. We can receive the change event.
3. We can store the resume token after each change event is called.
4. We can restart the application and resumeAfter the previous resume token.

For some unknown reason (maybe low activity due to weekend or overnight, or primary changes), the change stream stops receiving change events. I am not sure how to debug this issue as no event listeners are called on the change stream. ie. error, resumeTokenChanged, closed events are never ever triggered.

By chance after reviewing the mongoose code and noticing the returned changestream was a wrapped object, I decided to just use connection.getClient().watch which returns a mongdb native ChangeStream and I am able to successfully listen to resumeTokenChanged, closed, change and error events as I would have expected when using connection.watch().

When using this approach, I suprisingly found that the resumeTokenChanged event is called almost once per second and the token is changing. Also, the token returned by this event is also much shorter than the resume token that is provided in the change event.

I also noticed that if i stop the database (test failure) the change stream is closed (it calls the close event listener) and it is able to resume when i restart the database server.

My questions are -

1. Is connection.watch Mongoose Changestream object meant to output all the events?
2. Is it safe to use connection.getClient().watch to get a native change stream?
3. Why is resumeTokenChanged called every second, should I persist the token provided in this event?
4. In general, how can I detect if the change stream stopped working. I am wondering if the resumeTokenChanged event can be used as a heartbeat and it it stops for a period of time .. i should reconnect the changestream.

Sorry for the all the questions.
Regards,
Tarek