No retries are made after "failed to connect on first connect"

[joeytwiddle]

By default, mongoose throws an Error if the first connect fails, which crashes node.

So to reproduce this bug, you will need the following code in your app, to catch the error and prevent the crash:

db.on('error', console.error.bind(console, 'connection error:'));

Now we can reproduce this bug as follows:

1. Shut down your MongoDB
2. Start up your node app that uses mongoose
3. Your app will log: [MongoError: failed to connect to server [localhost:27017] on first connect [MongoError: connect ECONNREFUSED 127.0.0.1:27017]]
4. Start up your MongoDB again
5. Observe that mongoose does not now connect to the working MongoDB. The only way to get reconnected is to restart your app, or to use a manual workaround.

Expected behaviour: Since autoreconnect defaults to true, I would expect mongoose to establish a connection soon after the MongoDB is accessible again.

Note: If the first connect succeeds, but the connection to MongoDB is lost during runtime, then autoreconnect works fine, as expected. The problem is the inconsistency if MongoDB is not available when the app starts up.

(If this is the desired behaviour, and developers are recommended to handle this situation by not catching the error, and letting node crash, then I can accept that, but it is worth making it clear.)

node v4.4.1, mongoose@4.9.4, mongodb@2.2.19, mongodb-core@2.1.4

[TrejGun]

can confirm.
node 6, 7, mongoose (at leas 6 month), mongo 3.2 - 3.4
comes together with this one #4890

[sobafuchs]

i'm guessing this is a mongodb-core issue. It should attempt to reconnect even if the first try fails I think, since I'm not sure why that would be different from subsequent attempts.

Can you also report this issue there?

Here's a full repro script:

const mongoose = require('mongoose');
const co = require('co');
mongoose.Promise = global.Promise;
const GITHUB_ISSUE = `gh-5169`;


exec()
  .catch(error => {
    console.error(`Error: ${ error }\n${ error.stack }`);
  });


function exec() {
  return co(function*() {
    const db = mongoose.createConnection(`mongodb://localhost:27017/${ GITHUB_ISSUE }`);
    db.on('error', (error) => {
      console.error(`in an error ${ error }\n${ error.stack }`);
    })
  });
}

[joeytwiddle]

Thanks for the repro. I have looked into mongodb-core. It is the intended behaviour of the driver:

The driver will fail on first connect if it cannot connect to the host. This is by design to ensure quick failure on unreachable hosts. Reconnect behavior only kicks in once the driver has performed the initial connect.

It's up to the application to decide what to do. This is by design to ensure the driver fails fast and does not sit there making you think it's actually working.

So I suspect we won't get any different behaviour from the driver.

I actually think that behaviour is reasonable for a low-level driver. It will help developers who accidentally try to connect to the wrong host or the wrong port.

But if we want to do something more developer-friendly in mongoose, we could consider:

• When auto-reconnect options are enabled, keep trying to reconnect until the Mongo server can be contacted (by building in something like the workaround linked above).
• Log when mongoose is doing this, so in the case of a connection that never establishes, the developer will at least know where the problem is.
  (The log could be postponed, e.g. 30 seconds. Actually instead of direct logging, I guess we should emit an advisory error event, but still try the reconnect automatically.)

If I remember correctly, when I used the workaround and finally connected after a few failed attempts, queries already queued by the app did get executed as desired. (But this is worth testing again.)

[sobafuchs]

i think this is a decent idea actually, i'll label this a feature request

[vkarpov15]

No, failing fast on initial connection is a pretty consistent behavior across MongoDB drivers and there isn't much benefit to mongoose supporting it.

[bemosior]

This recent post from the Strongloop Loopback mongodb connector may be relevant. Their lazyConnect flag defers first connection until the endpoint is hit. If first connection fails in that case, the default connection loss settings will take effect (it will retry).

My interest is container orchestration, where "container startup order" can often be set and expected but "order of service availability" cannot. An orchestration tool might confirm that the mongo container is "up" even though the mongo service isn't available yet.

So, if my mongo container takes 1s to start but 5s for the service to become available, and my app container takes 1s to start and 1s for the service to be available, the app service will outrun the mongo service, causing a first connection failure as originally described.

The Docker Compose documentation has this to say:

Compose will not wait until a container is “ready” (whatever that means for your particular application) - only until it’s running. There’s a good reason for this.

The problem of waiting for a database (for example) to be ready is really just a subset of a much larger problem of distributed systems. In production, your database could become unavailable or move hosts at any time. Your application needs to be resilient to these types of failures.

To handle this, your application should attempt to re-establish a connection to the database after a failure. If the application retries the connection, it should eventually be able to connect to the database.

The best solution is to perform this check in your application code, both at startup and whenever a connection is lost for any reason.

So there's a definite gap here in the context of container orchestration, but both of these stances appear to be valid:

1. Mongoose could support an option to retry on first connect (perhaps defaulted to false with some cautionary documentation), or
2. Mongoose could place the responsibility on the developer to write code to retry if first connect fails.

[vkarpov15]

Sure, there's a gap, but then the responsibility falls to you to decide whether to retry if initial connection fails. All mongoose tells you is that it failed. If you make the questionable decision to use docker compose in production (or in any context for that matter), it's up to you to handle retrying initial connection failures.

[bemosior]

Stance 2, it is, then.

[joeytwiddle]

For anyone wanting auto-reconnection when first connect fails, this is how I handle it:

function createConnection (dbURL, options) {
    var db = mongoose.createConnection(dbURL, options);

    db.on('error', function (err) {
        // If first connect fails because mongod is down, try again later.
        // This is only needed for first connect, not for runtime reconnects.
        // See: https://github.com/Automattic/mongoose/issues/5169
        if (err.message && err.message.match(/failed to connect to server .* on first connect/)) {
            console.log(new Date(), String(err));

            // Wait for a bit, then try to connect again
            setTimeout(function () {
                console.log("Retrying first connect...");
                db.openUri(dbURL).catch(() => {});
                // Why the empty catch?
                // Well, errors thrown by db.open() will also be passed to .on('error'),
                // so we can handle them there, no need to log anything in the catch here.
                // But we still need this empty catch to avoid unhandled rejections.
            }, 20 * 1000);
        } else {
            // Some other error occurred.  Log it.
            console.error(new Date(), String(err));
        }
    });

    db.once('open', function () {
        console.log("Connection to db established.");
    });

    return db;
}

// Use it like
var db = createConnection('mongodb://...', options);
var User = db.model('User', userSchema);

For mongoose < 4.11 use db.open() instead of db.openUri()
For mongoose 4.11.7 this technique does not work.
For mongoose 4.13.4 it is working again!

---

Edit 2019/09/02: There is also a shorter solution using promiseRetry here.

[jorgecuesta]

Hi @vkarpov15 when this occurs also log an Unhandled Rejection from mongodb-core library.

MongoError: failed to connect to server [localhost:27017] on first connect [MongoError: connect ECONNREFUSED 127.0.0.1:27017]
    at Pool.<anonymous> (/Users/development/okkralabs/sem-server/services/sem-access/node_modules/mongodb-core/lib/topologies/server.js:336:35)
    at emitOne (events.js:116:13)
    at Pool.emit (events.js:211:7)
    at Connection.<anonymous> (/Users/development/okkralabs/sem-server/services/sem-access/node_modules/mongodb-core/lib/connection/pool.js:280:12)
    at Object.onceWrapper (events.js:317:30)
    at emitTwo (events.js:126:13)
    at Connection.emit (events.js:214:7)
    at Socket.<anonymous> (/Users/development/okkralabs/sem-server/services/sem-access/node_modules/mongodb-core/lib/connection/connection.js:187:49)
    at Object.onceWrapper (events.js:315:30)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at emitErrorNT (internal/streams/destroy.js:64:8)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

This is easy to reproduce, try to connect with a non available MongoDb server.

(node:2545) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): MongoError: failed to connect to server [localhost:27017] on first connect [MongoError: connect ECONNREFUSED 127.0.0.1:27017]
(node:2545) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Some way to handle it?

[vkarpov15]

@jorgearanda what version of mongoose are you using and can you provide some code samples?

[jorgearanda]

@vkarpov15 I think that's a message for @jorgecuesta ?

[vkarpov15]

Woops, my mistake. Silly github autocomplete, yep, that was for @jorgecuesta

[raythree]

I am seeing the exact same thing as @jorgecuesta, I was using 4.11.5 and but see the same with 5.0.0-rc2. I am using createConnection as some of the models are using different databases on the same mongo instance. It happens when starting the server while mongo is down:

2018-01-07 13:05:23-05:00: [INFO] database - reusing existing connectioun for mongodb://localhost/eiss
2018-01-07 13:05:23-05:00: [INFO] database - initializing database connection to: mongodb://localhost/eiss
2018-01-07 13:05:23-05:00: [INFO] database - reusing existing connectioun for mongodb://localhost/eiss
2018-01-07 13:05:23-05:00: [ERROR] database - Mongoose connection error: MongoNetworkError: failed to connect to server [localhost:27017] on first connect [MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017]
Unhandled rejection MongoNetworkError: failed to connect to server [localhost:27017] on first connect [MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017]
    at Pool.<anonymous> (/Users/bill/eiss4/js/node_modules/mongoose/node_modules/mongodb-core/lib/topologies/server.js:503:11)
    at emitOne (events.js:116:13)
    at Pool.emit (events.js:211:7)
    at Connection.<anonymous> (/Users/bill/eiss4/js/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:326:12)
    at Object.onceWrapper (events.js:317:30)
    at emitTwo (events.js:126:13)
    at Connection.emit (events.js:214:7)
    at Socket.<anonymous> (/Users/bill/eiss4/js/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connection.js:245:50)
    at Object.onceWrapper (events.js:315:30)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at emitErrorNT (internal/streams/destroy.js:64:8)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

The models call a connection function with the name of the database they are using (below). As long as mongo is running when the server starts, bringing the database up and down re-connects just fine (logging the errors). I just realized reading this issue that the driver handles the initial connection differently, which is kind of annoying. I am going to try @joeytwiddle's workaround, but I suspect this unhandled exception will still occur?

const allConnections = {};

module.exports = function(dbName) {
  const url = 'http://localhost/' + dbName;
  let conn;
  log.info('initializing database connection to: ' + url);

  conn = allConnections[url];
  if (!conn) {
    log.info('creating new connection for ' + url);
    conn = mongoose.createConnection(url, {
      useMongoClient: true,
      autoReconnect: true,
      autoIndex: false,
      reconnectTries: Number.MAX_SAFE_INTEGER
    });    
    // Log database connection events
    conn.on('connected', () => log.info('Mongoose connection open to ' + url));
    conn.on('error', (err) =>  log.error('Mongoose connection error: ' + err));
    conn.on('disconnected', () => log.error('Mongoose connection disconnected'));  
    allConnections[url] = conn;
  }
  else {
    log.info('reusing existing connection for ' + url);
  }
  return conn;
}

[vkarpov15]

@raythree in the above case you should be fine because you have .on('error'). If you want to retry initial connection if it fails, I'd recommend you just use async/await for that though

let conn;
for (let i = 0; i < numRetries; ++i) {
  try {
    conn = await mongoose.createConnection(uri);
    break;
  } catch (error) {}
}

[jorgecuesta]

@vkarpov15 Sorry for delay to answer you, we are using 4.11.14 and 4.13.4

[vkarpov15]

@jorgecuesta do mongoose.connect(uri).catch(err => {})

[jorgecuesta]

@vkarpov15 connect() return connection instance right?

const connection = mongoose.connect(uri || undefined, {useMongoClient: true});
connection.once('error', (e) => {
  console.error(e, 'mongoose connection error.');
});
connection.once('connected', () => {
  console.log('mongoose connected');
});

[vkarpov15]

In 5.0.0 we changed it so mongoose.connect() returns a promise consistently. In 4.x it returns a connection instance but with .then() and .catch() so you can use it with await

[vkarpov15]

@pranshuchittora can you show your MongoDB config file? Make sure the port is 27017. Also, try using 127.0.0.1 instead of localhost

[sitanshu-zymr]

Try to connect with ip instead of localhost that will fix your problem.

 let local = "mongodb://127.0.0.1:27017/XXX";
      mongoose.connect(
        local,
        { useNewUrlParser: true }
      );

[fider]

If first attempt to connect fails it always throws error.
You can bypass this using below (eg. when want to ensure that DB is running before you app/script start):

mongoose@5.3.16
@types/mongoose@5.3.5
@types/mongodb@3.1.17

async function waitForMongoDBStart(uri: string, timeoutMs: number) {

    return new Promise( async (resolve, reject) => {

        let endTime = Date.now() + timeoutMs;


        while (true) {

            let connectionError: Error;

            function errorHandler(err: Error) {
                connectionError = err;
            }
            mongoose.connection.once("error", errorHandler);

            await mongoose.connect(uri, {
                connectTimeoutMS: 5000, // This timeout applies only after connected & connection lost
                useNewUrlParser: true,
                useFindAndModify: false
            });

            // Time for error event propagation
            await wait(0);

            if ( ! connectionError) {
                mongoose.connection.removeListener("error", errorHandler);
                return resolve(); // All OK, connected
            }

            if (connectionError.name !== "MongoNetworkError") {
                return reject(`Unable to connect mongoDB. Details: ${connectionError}`);
            }

            if (Date.now() > endTime) {
                return reject(`Unable to connect mongoBD in ${timeoutMs} ms. Details: ${connectionError}`);
            }

        }

    });
}

[Jokero]

@vkarpov15 This is very non-obvious behaviour. From https://mongoosejs.com/docs/connections.html:

connectTimeoutMS - How long the MongoDB driver will wait 
before failing its initial connection attempt. 
Once Mongoose has successfully connected, connectTimeoutMS is no longer relevant.

Documentation says that it should work with initial connection, but it does not. With failed to connect to server error it fails immediately.

[Jokero]

If it's consistent behaviour across MongoDB drivers, it would be nice to have some option like reconnectOnInitialFail which will be false by default

[fider]

Current behavior is opposite to this one described by vkarpov15 - connectImeoutMS matters only after first connection (on first connection it is expected that DB is started and accepting connection, otherwise immedia error is thrown) sob., 22 gru 2018 o 13:03 Dmitry Kirilyuk <notifications@github.com> napisał(a):

…

If it's consistent behaviour across MongoDB drivers, it would be nice to have some option like reconnectOnInitialFail which will be false by default — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5169 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABY-TjgeI2UqVca050y5YY3zi6w7nMkfks5u7h-vgaJpZM4M-1ur> .

-- *Wojciech Fiderek* Mobile: +48 516 661 428

[vkarpov15]

If I recall correctly, connectTimeoutMS matters on initial connection depending on network conditions and os. When does connectTimeoutMS matter after initial connection?

[fider]

In my case immediate error was thrown when I was not able to connect on first try to localhost mongod instance on Windows 10 and Alpine Linux (version unknown). Mongod instance just not yet started.
Other examples than localhost? Anyone?

[vkarpov15]

@fider @Jokero I dug in further, and I'm right, connectTimeoutMS only matters until you've made an initial connection, socketTimeoutMS takes over afterward. Here's the relevant code in the MongoDB driver:

• https://github.com/mongodb-js/mongodb-core/blob/177e558be28e5505ce1eb4a94b35d05130d34e76/lib/connection/connection.js#L599
• https://github.com/mongodb-js/mongodb-core/blob/177e558be28e5505ce1eb4a94b35d05130d34e76/lib/connection/connection.js#L604

connectTimeoutMS isn't always useful because you'll usually get a DNS error or a connection refused error. But if you, say, open up a TCP server that listens on a port but doesn't actually do anything:

const net = require('net');

const server = net.createServer();

server.listen(27055);

You'll see that connectTimeoutMS kicks in:

const assert = require('assert');
const mongoose = require('mongoose');
mongoose.set('debug', true);

const { Schema } = mongoose;

run().then(() => console.log('done')).catch(error => console.error(error.stack));

async function run() {
  await mongoose.connect('mongodb://localhost:27055', {
    useNewUrlParser: true,
    connectTimeoutMS: 1000,
    socketTimeoutMS: 25000
  });
} 

$ time node gh-5169.js 
MongoNetworkError: connection 0 to localhost:27055 timed out
    at Socket.<anonymous> (/mongoose/node_modules/mongodb-core/lib/connection/connection.js:259:7)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:106:13)
    at Socket.emit (events.js:208:7)
    at Socket._onTimeout (net.js:407:8)
    at ontimeout (timers.js:475:11)
    at tryOnTimeout (timers.js:310:5)
    at Timer.listOnTimeout (timers.js:270:5)

real	0m2.293s
user	0m0.271s
sys	0m0.043s
$ 

[keermanish]

sometimes due to network restrictions, MongoDB url's gets blocked, try changing your network/internet source.

https://stackoverflow.com/questions/47958683/cannot-connect-to-mongodb-atlas-through-mongo-shell?answertab=active#tab-top

[joeytwiddle]

When making an initial connection, (at least) three things can happen:

1. There is a mongodb and the connection succeeds, yay!

2. The server responds "Connection refused" which means there is no process listening on this port, or there is a firewall which actively rejected the connection. (This usually happens immediately, and I think this is what fider was experiencing.)

3. The server does not respond at all, e.g. the firewall passively dropped the packets or there is no server at that IP address. (In this case, vkarpov's connectTimeoutMS is eventually triggered.)

So the timeout is not always used. It's only used if there is no clear success or failure in the given time.

This is common in networking, and also when talking to an uncooperative person. Your request can get two types of rejection: "NO!" and .....

[vkarpov15]

Thanks for the solid explanation @joeytwiddle 👍

[esetnik]

How should one handle this behavior with replica set connections using mongodb+srv? When my replica set is in a rolling restart scenario (database update) I start getting errors like connect ECONNREFUSED 34.238.xxx.xxx:27017.

[vkarpov15]

@esetnik are you using useUnifiedTopology: true and Mongoose 5.7.5? We identified an issue #8209 that might be related.

[esetnik]

@vkarpov15 yes I am! Thanks for the tip. I will run some tests after disabling unified topology and report back my findings.

[fdmxfarhan]

Hi. I have the exact same problem in my project. have you found any soloution for it?!

[vkarpov15]

@fdmxfarhan please open a separate issue and follow the issue template.