MongoDB Integration causes intermittent connection errors (LAMBDA_RUNTIME fails)

[Nikhil22]

Discussed in #10621

^{Originally posted by Nikhil22 September 28, 2023}

Question

When I connect to MongoDB (via MONGODB_URI env variable, via the node driver) from my Vercel-deployed app, I get intermittent errors (MongoServerSelectionError) that present as 500 errors in my Vercel-deployed app. The errors go away on refresh.

An example of the error is:

ERROR	Unhandled Promise Rejection 

{
   "errorType":"Runtime.UnhandledPromiseRejection",
   "errorMessage":"MongoServerSelectionError: Server selection timed out after 30000 ms",
   "reason":{
      "errorType":"MongoServerSelectionError",
      "errorMessage":"Server selection timed out after 30000 ms",
      "reason":{
         "type":"ReplicaSetNoPrimary",
         "servers":{
            
         },
         "stale":false,
         "compatible":true,
         "heartbeatFrequencyMS":10000,
         "localThresholdMS":15,
         "setName":"atlas-j7739j-shard-0",
         "maxElectionId":null,
         "maxSetVersion":null,
         "commonWireVersion":0,
         "logicalSessionTimeoutMinutes":null
      },
      "stack":[
         "MongoServerSelectionError: Server selection timed out after 30000 ms",
         "    at Timeout._onTimeout (/var/task/node_modules/mongodb/lib/sdam/topology.js:278:38)",
         "    at listOnTimeout (node:internal/timers:569:17)",
         "    at process.processTimers (node:internal/timers:512:7)"
      ]
   },
   "promise":{
      
   },
   "stack":[
      "Runtime.UnhandledPromiseRejection: MongoServerSelectionError: Server selection timed out after 30000 ms",
      "    at process.<anonymous> (file:///var/runtime/index.mjs:1250:17)",
      "    at process.emit (node:events:526:35)",
      "    at emit (node:internal/process/promises:149:20)",
      "    at processPromiseRejections (node:internal/process/promises:283:27)",
      "    at process.processTicksAndRejections (node:internal/process/task_queues:96:32)"
   ]
}

"LAMBDA_RUNTIME Failed to post handler success response. Http response code":"400.

Just to be clear, the connection string and credentials work and are correct as it does connect properly when errors are not there, but when errors that I describe above do occur, it crashes the entire app.

I've written about this in detail in the MongoDB community here (https://www.mongodb.com/community/forums/t/no-way-to-avoid-replicasetnoprimary-errors/243498).

I've also been in contact with MongoDB support and they said that there are no problems with the Mongo cluster itself.

Moreover, upon further research, it seems that this is a common problem when trying to connect to MongoDB from Vercel. For example, here (https://www.mongodb.com/community/forums/t/intermittently-getting-mongoserverselectionerror-error-on-my-next-js-vercel-deployment/242470).

I'd like to solve this problem, or at the very least, not have my entire app crash with a 500 due to the error, as it currently does.

Please help.

[bstchow]

I am also encountering a nearly identical issue. Strangely enough the timeouts seem to occur almost instantaneously, suggesting there is some sort of stale connection or the issues are being misrepresented as timeouts.

[niudai]

Encountering the same issue!

[apsquared]

Same issue here. A lot of other articles on the web talk about connection count issues, but my connection count is never above 2 while this is happening. I think its actually the opposite, some type of stale connection.

[BreakStateDev]

I'm experiencing the same issue, with no clear pattern as when the issue occurs...

[apsquared]

Yea, it's really tough. I've done a ton of debugging and I really think it's outside the developers control. When the issue happens NO logging in my code ever runs, even things that are output before calls to the DB code. My best guess:

A previous call leaves a connection open, during the management of that connection something goes wrong and throws an unhandled exception, when a future request is handled by the same thread it already has an unhandled exception waiting for it.

I have no real evidence of this but its the only thing that makes sense.

Also, monitoring the events I see the serverClosedEvent happens almost at the same time as the error.

[DeeK-Dev]

+1 to this. same issue.

[life-efficient]

Same issue

[shako]

Same issue here. Is there any news on this, or hope?

[raulmarindev]

In my case, it's failing when next-auth tries to refetch the session. The request to mongo fails and the session is lost.

[Larry-Osakwe]

Happens to me as well. Any fix for this?

[Saswat689]

Vercel FIX THIS!

[arushbansal]

Same problem. Please fix this

[Rieranthony]

I've got the same issue, here is a full log:
Unhandled Promise Rejection {"errorType":"Runtime.UnhandledPromiseRejection","errorMessage":"MongoNetworkTimeoutError: connection timed out","reason":{"errorType":"MongoNetworkTimeoutError","errorMessage":"connection timed out","connectionGeneration":0,"stack":["MongoNetworkTimeoutError: connection timed out"," at connectionFailureError (/var/task/node_modules/.pnpm/mongodb@6.3.0/node_modules/mongodb/lib/cmap/connect.js:381:20)"," at TLSSocket.<anonymous> (/var/task/node_modules/.pnpm/mongodb@6.3.0/node_modules/mongodb/lib/cmap/connect.js:285:22)"," at Object.onceWrapper (node:events:631:28)"," at TLSSocket.emit (node:events:517:28)"," at TLSSocket.emit (node:domain:489:12)"," at Socket._onTimeout (node:net:598:8)"," at listOnTimeout (node:internal/timers:569:17)"," at process.processTimers (node:internal/timers:512:7)"]},"promise":{},"stack":["Runtime.UnhandledPromiseRejection: MongoNetworkTimeoutError: connection timed out"," at process.<anonymous> (file:///var/runtime/index.mjs:1276:17)"," at process.emit (node:events:529:35)"," at process.emit (node:domain:489:12)"," at emit (node:internal/process/promises:149:20)"," at processPromiseRejections (node:internal/process/promises:283:27)"," at process.processTicksAndRejections (node:internal/process/task_queues:96:32)"]} Error: Runtime exited with error: exit status 128 (Unhandled Promise Rejection)

It appears at random and throws 500 errors, it seems that it doesn't follow a definitive pattern.

I'm also using Prisma as ORM, but it doesn't seems to be linked to that.

I found some clues about people having the same issue on AWS Lambda.

[oliveiracdz]

Same problem here.
Occasionally, the communication with MongoDB simply crashes and we need to restart the application on Vercel (PRO).

Please, the community need a workaround at least.

[auspy]

yup same issue

[Larry-Osakwe]

PLEASE FIX

[fsultani]

Same thing happening here, and my app is in production. Not a good UX. I hope they release a fix ASAP.

[atifrc]

Same issue here

[smaeda-ks]

As a general best practice, please have a look at their official guide here:
https://www.mongodb.com/docs/atlas/manage-connections-aws-lambda/

Note: for the callbackWaitsForEmptyEventLoop setting, we have an equivalent setting applied to all customers by default so don't worry about that. But other stuff, such as:

• Define the client to the MongoDB server outside the AWS Lambda handler function (i.e., use module cache/singleton pattern)
• Tunning maxIdleTimeMS parameter (important! Set it properly based on your plan's max timeout limit - docs)
• etc

are still very relevant to this type of error. So be sure to test them out as well. This is not necessarily an issue with the Vercel platform itself, but more of a common issue when you're dealing with raw TCP connections on serverless platforms in general.

I wish Mongo team has an HTTP-based client library that utilizes their REST APIs:
https://www.mongodb.com/docs/atlas/app-services/data-api/#when-to-use-the-data-api

[smaeda-ks]

@fsultani @Nikhil22 could you please test with the latest driver?
https://github.com/mongodb/node-mongodb-native/releases/tag/v6.3.0

from the mongo team:

This version introduced automatic server monitoring strategy selection for Lambda (see NODE-5197), so it's possible part of the problem here could be with runtime environment's thawing and having a stale view of the topology (which is why the error shows ReplicaSetNoPrimary and no servers.

[apsquared]

I upgraded to 6.3.0, but am still getting the UnhandledPromiseRejection error in my logs.

[fsultani]

The only guide I found for the integration with MongoDB was here. It does a good job over all, but it's a good practice to always return a response for an error, which they don't do.

This is also a good reference to read: https://www.mongodb.com/docs/drivers/node/current/faq/

I managed to almost eliminate the timeout errors by making a few adjustments. For one, add these options to increase the timeout value:

const options = {
  family: 4,
  connectTimeoutMS: 60000,
  serverSelectionTimeoutMS: 60000,
};

The official docs for family say it's no longer supported, so you can probably leave it out.

The one change that has probably helped the most was retrying the connection if there's an error. I no longer see numerous connection timeouts in my server logs the way I did before.

Here's my full connection code:

import { MongoClient } from 'mongodb';

if (!process.env.MONGODB_URI) {
  throw new Error('Invalid/Missing environment variable: "MONGODB_URI"');
}

const uri = process.env.MONGODB_URI;
const options = {
  family: 4,
  connectTimeoutMS: 60000,
  serverSelectionTimeoutMS: 60000,
};

let client;
let clientPromise;

let attempt = 1;
const retries = 2;

try {
  if (process.env.NODE_ENV === "development") {
    // In development mode, use a global variable so that the value
    // is preserved across module reloads caused by HMR (Hot Module Replacement).
    let globalWithMongo = global;

    if (!globalWithMongo._mongoClientPromise) {
      client = new MongoClient(uri, options);
      globalWithMongo._mongoClientPromise = client.connect();
    }
    clientPromise = globalWithMongo._mongoClientPromise;
  } else {
    // In production mode, it's best to not use a global variable.
    client = new MongoClient(uri, options);
    clientPromise = client.connect();
    console.log(`Successfully connected after ${attempt} ${attempt > 1 ? 'attempts' : 'attempt'}`);
  }
} catch (error) {
  console.log(`error\n`, error);
  console.log(`Trying again...`);

  if (attempt < retries) {
    attempt += 1;
    client = new MongoClient(uri, options);
    clientPromise = client.connect();
    console.log(`Successfully connected after ${attempt} ${attempt > 1 ? 'attempts' : 'attempt'}`);
  }
}

// Export a module-scoped MongoClient promise. By doing this in a
// separate module, the client can be shared across functions.
export default clientPromise;

I got the majority of this code from another article.

I still get erros from time to time, but not nearly as often, and my server logs don't show connection timeout errors nearly as frequently. I hope this helps others.

[JWPapi]

Thank you for your work @fsultani

I digged a bit deeper. So for me this is connected to next-auth. Client keeps calling /api/auth/session every couple minutes. One time after 22 minutes it failed. I implemented @fsultani’s solution.

For me the options were new and the reconnecting logic.

I tried to keep a tab open again and 40 minutes long no error happened, which made me hopeful. Unfortunately today I could see that the error happened to more users, not sure if less often than before.

Unfortunately I also couldn’t see any notice of a successful connection after 2 tries. So I have no evidence the reconnect even works. I would suggest trying more than 1 reconnect.

The weird thing for me is that I’m not even using the MongoDBAdapter as next-auth suggests, only JWT Auth. So the session request shouldn’t even call the db. However it still throws that error.

I could try using the adapter, but even the next-auth mongodb adapter file says:

The MongoDB adapter does not handle connections automatically, so you will have to make sure that you pass the Adapter a `MongoClient` that is connected already. Below you can see an example how to do this.

So I assume so far a configuration in that file, or really Vercel has to fix this.

[weineran]

I am using mongoose to connect to MongoDB from my Vercel NextJS app. I was getting the same MongoServerSelectionError[....]ReplicaSetNoPrimary error above.

After a lot of searching, implementing retries, scratching my head, I found and implemented the lambda pattern at the bottom of this page: https://mongoosejs.com/docs/lambda.html

I haven't seen the error since then. I think it's resolved? 🤞

[steve-torres]

@weineran any chance you could share your mongo connection code with the community? I'd love to compare

[weineran]

Here it is. I should note that I have retries in there, but from what I could tell those didn't help much if at all. Once I added the global let conn, that's what seemed to resolve the problem.

import mongoose from "mongoose";
import User from "@/models/User";

let conn: Promise<mongoose.Mongoose> = null;

// NOTES(AW)
// * In Feb 2024, I was getting quite a few errors like this in prod: "MongoServerSelectionError: Server selection timed out after 30000 ms [...] ReplicaSetNoPrimary"
// * See here: https://github.com/vercel/vercel/discussions/11178
// * See also: https://discord.com/channels/1152132099499905086/1204426230871367701/1204426230871367701
// * For now, I am addressing this with retries
// * 3/1/24 12:35 AM: Still failing with retries. Need to try the lambda pattern here: https://mongoosejs.com/docs/lambda.html
// * 3/1/24 9:32 AM: Now following the lambda pattern here: https://mongoosejs.com/docs/lambda.html

const connectMongo = async () => {
  if (!process.env.MONGODB_URI) {
    throw new Error(
      "Add the MONGODB_URI environment variable inside .env.local to use mongoose"
    );
  }

  // If the connection already exists, return it
  if (conn != null) {
    return conn;
  }

  // Otherwise the connection is null. So try to connect, with retries.
  const maxTries = 3;

  const options = { heartbeatFrequencyMS: 15000 };

  let bubbleError = [];

  for (let i = 0; i < maxTries; i++) {
    try {
      conn = mongoose
        .connect(process.env.MONGODB_URI, options)
        .then(() => mongoose);

      // `await`ing connection after assigning to the `conn` variable
      // to avoid multiple function calls creating new connections
      await conn;

      return conn;
    } catch (e) {
      e.message = `${i + 1}: Mongoose Client Error: ${e.message}`;
      console.warn(`${e.message}`);
      bubbleError.push(e);
    }
  }

  throw new Error(
    `DB connection failed [${maxTries}] times: ${bubbleError.join(" | ")}`
  );
};

export default connectMongo;

I should mention that my website (harvestiy.com) has very little traffic. But I was noticing the error a lot during alpha testing, and I haven't noticed the error at all since adopting the lambda pattern from the mongoose docs.

[weineran]

Sad news: I just got the MongoServerSelectionError[....]ReplicaSetNoPrimary error again. So my fix above is not a real fix. Does it even help? Hard to say...I think it seems to have helped reduce the issue, but I don't have any real data to support that belief.

[JWPapi]

I’m trying this config for mongodb instead of mongoose, but it didn’t work

import { MongoClient } from 'mongodb'

if (!process.env.MONGODB_URI) {
  throw new Error('Invalid/Missing environment variable: "MONGODB_URI"')
}

const uri = process.env.MONGODB_URI
const options = {
  connectTimeoutMS: 60000,
  maxIdleTimeMS: 60000,
  serverSelectionTimeoutMS: 60000
}

let client
let clientPromise

async function connectToDatabase() {
  if (!client) {
    client = new MongoClient(uri, options)
    await client.connect()
    console.log(`Successfully connected to the database.`)
  }
  return client
}

if (process.env.NODE_ENV === 'development') {
  // In development mode, use a global variable so that the value
  // is preserved across module reloads caused by HMR (Hot Module Replacement).
  if (!global._mongoClient) {
    global._mongoClient = connectToDatabase()
  }
  clientPromise = global._mongoClient
} else {
  // In production mode, directly attempt to connect.
  clientPromise = connectToDatabase()
}

// Export the clientPromise for use in other parts of your application
export { clientPromise }

// Provide a helper function to get a collection from the database
export async function getCollection(collectionName) {
  const client = await clientPromise
  return client.db('deeplead').collection(collectionName)
}

I’m not using models, but I’m gonna try your mongoose version now

[ceduch]

Update your mongodb driver to 6.5, in the release description they mentionned "Fixed memory leak in Connection layer: The Connection class has recently been refactored to operate on our socket operations using promises. An oversight how we made async network operations interruptible made new promises for every operation. We've simplified the approach and corrected the leak."

That did the trick for me

[Rieranthony]

Anyone found a solution to this using Prisma + MongoDB?

[IlyyA]

So don't call connect() at all and just wrap the client in a promise and return it? How long has that worked for you? How many times were you getting the error before implementing this and how many times after? Any other side-effects?

import { MongoClient, ServerApiVersion } from "mongodb";

if (!process.env.MONGODB_URI) {
  throw new Error('Invalid/Missing environment variable: "MONGODB_URI"');
}

const uri = process.env.MONGODB_URI;
const options = {
  connectTimeoutMS: 60000,
  maxIdleTimeMS: 60000,
  serverSelectionTimeoutMS: 60000,
};
let cachedPromise = global._mongoClientPromise;
let cachedClient = global._mongoClient;
class Singleton {
  constructor() {
    if (cachedPromise && cachedClient) {
      console.log("In Global");
      this.client = cachedClient;
      this.clientPromise = cachedPromise;
    } else {
      console.log("No Global");
      this.client = global._mongoClient = new MongoClient(uri, options);
      this.clientPromise = global._mongoClientPromise = this.client.connect();
    }
  }

  static get instance() {
    if (!this._instance) {
      this._instance = new Singleton();
    }
    return this._instance.clientPromise;
  }
}

Singleton._instance = null;

const clientPromise = Singleton.instance;

// Export a module-scoped MongoClient promise. By doing this in a
// separate module, the client can be shared across functions.
export default clientPromise;

So this will turn into:

import { MongoClient, ServerApiVersion } from "mongodb";

if (!process.env.MONGODB_URI) {
  throw new Error('Invalid/Missing environment variable: "MONGODB_URI"');
}

const uri = process.env.MONGODB_URI;
const options = {
  connectTimeoutMS: 60000,
  maxIdleTimeMS: 60000,
  serverSelectionTimeoutMS: 60000,
};
let cachedPromise = global._mongoClientPromise;
let cachedClient = global._mongoClient;
class Singleton {
  constructor() {
    if (cachedPromise && cachedClient) {
      console.log("In Global");
      this.client = cachedClient;
      this.clientPromise = cachedPromise;
    } else {
      console.log("No Global");
      this.client = global._mongoClient = new MongoClient(uri, options);
      this.clientPromise = global._mongoClientPromise = new Promise((resolve) => { resolve(this.client) });
    }
  }

  static get instance() {
    if (!this._instance) {
      this._instance = new Singleton();
    }
    return this._instance.clientPromise;
  }
}

Singleton._instance = null;

const clientPromise = Singleton.instance;

// Export a module-scoped MongoClient promise. By doing this in a
// separate module, the client can be shared across functions.
export default clientPromise;

[ceduch]

Yes just like that, I never got the error since, and before I use to get it randomly couple times a day depending on traffic.

[ceduch]

If you want to understand what's going on with the client try listening to connection events. To follow every client I generated a clientId and the timestamp of the file init so I could know for how long it was running before it lost the connection.

const uri = process.env.MONGODB_URI;
const clientId = generateId(4);
const initiatedDt = Date.now();

After that I setted the listeners on the client.

mongoClient.on("connectionClosed", (event) => { 
    const since = secondsToDurationString(differenceInSeconds(Date.now(), initiatedDt));
    console.log(`[MONGO] [${clientId}] [${since}] Connection closed`, JSON.stringify(event)); })
mongoClient.on("connectionPoolCleared", (event) => { 
    const since = secondsToDurationString(differenceInSeconds(Date.now(), initiatedDt));
    console.log(`[MONGO] [${clientId}] [${since}] Connection pool cleared`, JSON.stringify(event)); })
mongoClient.on("connectionPoolCreated", (event) => { 
    const since = secondsToDurationString(differenceInSeconds(Date.now(), initiatedDt));
    console.log(`[MONGO] [${clientId}] [${since}] Connection pool created`, JSON.stringify(event)); })
mongoClient.on("connectionPoolClosed", (event) => { 
    const since = secondsToDurationString(differenceInSeconds(Date.now(), initiatedDt));
    console.log(`[MONGO] [${clientId}] [${since}] Connection pool closed`, JSON.stringify(event)); })
mongoClient.on("connectionCreated", (event) => { 
    const since = secondsToDurationString(differenceInSeconds(Date.now(), initiatedDt));
    console.log(`[MONGO] [${clientId}] [${since}] Connection created`, JSON.stringify(event)); })

Try that using the .connect() method, and then the other way with the client directly, you will understand what is going on. If you can't reproduce the bug locally, try making a server rendered page that prefetch data using the mongo client and try refreshing the page a couple of times until it does it (shouldn't take long). Follow the logs in your terminal while doing that. If you have a dev environment on vercel you can also try leaving the logs to see what happend before getting the lost connection or whatever error you get.

[maulikdhameliya]

No idea for Prisma, but what did the trick for me was using the new MongoClient() directly and not mongoClient.connect().

I am wondering has anyone tried the solution given by @ceduch @IlyyA

@ceduch how long has it been since you implemented this solution and no error is coming?

[ceduch]

It's been a couple weeks since I've implemented it in production, never had another problem with my mongodb client since, and it used to happen daily.

[aldosch]

Hi everyone.
We have submitted a node-mongodb-native bug report to the MongoDB team: NODE-6179

It references this Github issue/discussion. If you could share any additional context or troubleshooting attempts there it would be very helpful. I understand this issue has been affecting many teams and we really appreciate your patience.

Thanks,
Aldo