How to properly handle watcher errors/disconnection

[Apokalypt]

Prerequisites

• I have written a descriptive issue title

Mongoose version

8.0.3

Node.js version

20.10.0

MongoDB version

6.0.13 (atlas server) & 6.2.0 (node)

Operating system

Linux

Operating system version (i.e. 20.04, 11.3, 10)

Debian Bullseye 11

Issue

Hello, for some time now, some of my watchers have been disconnecting and not reconnecting, resulting in events not being received. I'm unable to reproduce the problem effectively locally, but once in production it often happens:

1. I start my NodeJS service
2. I start my watchers
3. I log each insertion event
4. After a while (sometimes 1 month or 10 days or less) I don't receive any more insertion events even though I'm creating documents.

Here is my current code :

const MyMongooseModel = mongoose.model('ModelName', mySchema);

MyMongooseModel.watch([{ $match: { operationType: "insert" } }])
     .on("change", async data => {
          console.log(data);
      })
      .on("error", error => {
            console.error("Error on log watch stream", error);
      });

--> How can I efficiently detect when watchers are completely closed in order to restart it ?

N.B. I've tried listening for the "close" event but sometimes it's sent but events are always received and sometimes afterwards there are no more events received.

[github-actions]

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 5 days

[Apokalypt]

After analyzing the problem a little more closely, I realized several things:

• there are no "end" events after a "close" or an "error" (is this the expected behavior?)
• "close" is emitted when the connection is closed, but there may be an automatic re-connection attempted immediately afterwards
• "error" is sent when the connection is closed due to an error

So, simply call the ".watch(...)" method when the "error" event is emitted, but ignore the others in order to keep watchers open.
This leaves me wondering two things:

• Why not emit the "end" event when the stream is finished and no reconnection is made?
• Why not propose, via the options of the ".watch(...)" method, an automatic reconnection even in the event of an error?

[vkarpov15]

Based on our investigation from #14419, 'close' is the event to listen for. In Node.js streams, 'close' means the stream is closed and no more events should be emitted. We did find a minor issue where the MongoDB Node driver emits duplicate 'close' events, but when we run the following script it looks like there's no more non-'close' events emitted after 'close' if you stop and restart the underlying replica set:

'use strict';
  
const mongoose = require('mongoose');

void async function main() {
  const uri = 'mongodb://127.0.0.1:27017,127.0.0.1:27018/mongoose_test';
  await mongoose.connect(uri);
  const Person = mongoose.model('Person', mongoose.Schema({ name: String }));
  await Person.createCollection();

  let resumeAfter = null;
  let changeStream = await Person.watch();
  addChangeStreamListeners(changeStream);

  function addChangeStreamListeners(changeStream) {
    changeStream.on('change', data => console.log(new Date(), data));
    changeStream.on('resumeTokenChanged', data => {
      console.log(new Date(), 'resumeTokenChanged', data);
      resumeAfter = data;
    });
    changeStream.on('error', data => console.log(new Date(), 'error', data));
    changeStream.on('close', data => {
      console.log(new Date(), 'close', data, new Error().stack);
      //changeStream = Person.watch([], { resumeAfter });
      //addChangeStreamListeners(changeStream);
    });
  }

  while (true) {
    await new Promise(resolve => setTimeout(resolve, 8000));
    // Insert a doc, will trigger the change stream handler above
    console.log(new Date(), 'Inserting doc');
    await Person.create({ name: 'Axl Rose' });
    console.log(new Date(), 'Inserted doc');
  }
}();

Why not emit the "end" event when the stream is finished and no reconnection is made?: Node.js stream API doesn't specify an 'end' event. There's a 'finish' event, but that's when end() is explicitly called. close is the more correct event to listen for

Why not propose, via the options of the ".watch(...)" method, an automatic reconnection even in the event of an error? We're planning on implementing that in #14419