Read ECONNRESET every 3 mins - NodeJS server crashes due to this. Using newrelic plugin for server monitoring.

[ayazpasha2434]

Error: Redis connection to redis1.tfs.tfs:6379 failed - read ECONNRESET at RedisClient.on_error (/var/www/a/b/c/node_modules/redis/index.js:196:24) at Socket. (/var/www/a/b/c/node_modules/redis/index.js:106:14)
at Socket.emit (events.js:95:17)
at net.js:441:14
at /var/www/a/b/c/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/glue.js:188:31
at /var/www/a/b/c/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/glue.js:188:31
at process._tickDomainCallback (node.js:486:13)
at process. (/var/www/a/b/c/node_modules/newrelic/node_modules/continuation-local-storage/node_modules/async-listener/index.js:18:15)

[brycebaril]

To me, this sounds like a networking issue outside Node.js, this error "ECONNRESET" is when the remote connection to Node.js closes the socket unexpectedly.

Are you using a proxy between Node and Redis?

What version of Node/node_redis/Redis server?

I doubt the NewRelic agent is involved, but you could always try to reproduce without the agent and see if that fixes the issue. cc/ @wraithan

[wraithan]

Those lines are just CLS doing it's thing. With more of a repro I could fully rule out the New Relic agent. But I'm relatively confident this isn't being caused by us.

[ayazpasha2434]

I removed the NewRelic agent, the problem still occurs. Approx. every 3 mins I get this crash happening. Below are the logs for reference.

events.js:72
throw er; // Unhandled 'error' event
^
Error: Redis connection to redis1.tfs.tfs:6379 failed - read ECONNRESET
at RedisClient.on_error (/var/www/a/b/c/node_modules/redis/index.js:196:24)
at Socket. (/var/www/a/b/c/node_modules/redis/index.js:106:14)
at Socket.emit (events.js:95:17)
at net.js:441:14
at process._tickDomainCallback (node.js:486:13)
error: Forever detected script exited with code: 8
error: Script restart attempt #50

Also, I'm using Node.JS - v0.10.35, node_redis - 0.12.1 & Redis - 2.8.4.
I'm accessing Redis directly via IP in Node.

[sarathms]

I was facing this problem too, but just got resolved after I realized I was playing around with my /etc/hosts file yesterday for some other reason. It was missing:

127.0.0.1 localhost

@ayazpasha2434 the solution might not be the same for you, but like others are hinting, it is defintely outside node and in the env/shell/OS/VM. Step back a little and stare at it.

[ayazpasha2434]

@sarathms I have this mapping in my hosts file. However, the Redis machine in my case is a remote one. Just cross-checked, I have the said localhost mapping in Redis machine as well. Still the problem is not solved.

[Waterloo]

I am also facing the same problem . Here is the Console Output

events.js:72
throw er; // Unhandled 'error' event
^
Error: Redis connection to pub-redis-10606.us-east-1-1.1.ec2.garantiadata.com:10606 failed - read ECONNRESET at RedisClient.on_error (C:\Users\Gigi\Desktop\eduknow_api_ver_1\auth\node_m
odules\redis\index.js:196:24)
at Socket. (C:\Users\Gigi\Desktop\eduknow_api_ver_1\auth\node_modules\redis\index.js:106:14)
at Socket.emit (events.js:95:17)
at net.js:441:14
at process._tickCallback (node.js:442:13)

[Waterloo]

@ayazpasha2434 This line solved my error

client.on('error',function(err){ console.error(err)})

[blainsmith]

@ayazpasha2434 Did @Waterloo have the right answer for you?

[BridgeAR]

There seemed to be an issue with your network connection and it does not seem to be an issue with node redis. Adding the error handler should be sufficient so the app does not crash anymore (and it should always be attached).

[biggu0]

I met this problem when I post a http(use post) to JAVA server.{ [Error: read ECONNRESET]
code: 'ECONNRESET',
errno: 'ECONNRESET',
syscall: 'read',
response: undefined }
Error: read ECONNRESET
at exports._errnoException (util.js:856:11)
at TCP.onread (net.js:546:26)
double callback!
POST /user_archives/create/save 500 45.306 ms - 386
Error: read ECONNRESET
at exports._errnoException (util.js:856:11)
at TCP.onread (net.js:546:26)

[BridgeAR]

@biggu0 I'm not sure what connection this has to node_redis. To find out more about ECONNRESET you might want to look at stackoverflow.

[akhare22sandeep]

Hi There ,
We are also seeing this issue. We observe that it could be because of redis SNAPSHOTTING . Whenever redis is taking snapshot and storing on the disk connection is getting reset. To prove myself i compare the application logs with redis debug logs and the timings are matching exactly.

below are the snapshotting setting in redis.conf
save 900 1
save 300 10
save 60 10000

Please let us know if we are doing something wrong.

[saranyapm]

I am also facing connection issue frequently .

ERR at RedisClient.emit (events.js:188:7)
ERR at RedisClient.on_error (/home/vcap/app/node_modules/redis/index.js:394:14)
ERR at RedisClient. (/home/vcap/app/server.js:28:13)
ERR at emitErrorNT (net.js:1281:8)
ERR at Socket. (/home/vcap/app/node_modules/redis/index.js:272:14)
ERR at process._tickCallback (internal/process/next_tick.js:98:9)
ERR Error: Redis connection to XXXXXXXX:XXXXX failed - read ECONNRESET
ERR at TCP.onread (net.js:572:26)
ERR at _combinedTickCallback (internal/process/next_tick.js:74:11)
ERR at emitOne (events.js:96:13)
ERR at exports._errnoException (util.js:1022:11)
ERR at Socket.emit (events.js:188:7)

After getting the error it is automatically reconnecting.
What can be the issue to lost the connection.

[sw-tt-tarunvarma]

Hi Waterloo ,
Where you put this statement client.on('error',function(err){ console.error(err)})
and i think this statement just for console your error nothing else.
is it really usable?
Please give me your thoughts asap.

Thanks in advance!!!

[riginoommen]

Hello

I am also facing the same issue

events.js:160
throw er; // Unhandled 'error' event
^

Error: read ECONNRESET

[hametGh]

This solved my problem with:

1. I disabled all of my processes monitoring (new relic, elastic apm ...)
2. I used this code

process.on('uncaughtException', function (err) {
 console.error(err.stack);
 console.log("Node NOT Exiting...");
});

[stormit-vn]

I ma also facing this issue. I am using with sails.js framework with a hook, so there is only one connection to redis server when starting node service, we will use global sails object to access and work with redis

redisClient = redis.createClient({
    host: redisConfig.host,
    port: redisConfig.port,
    db: redisConfig.db,
    retry_strategy: (retryOptions) => {
        if (retryOptions.error && retryOptions.error.code === 'ECONNREFUSED') {
            // End reconnecting on a specific error and flush all commands with
            // a individual error
            return new Error('The server refused the connection');
        }

        if (retryOptions.total_retry_time > 1000 * 60 * 60) {
            // End reconnecting after a specific timeout and flush all commands
            // with a individual error
            return new Error('Retry time exhausted');
        }

        if (retryOptions.attempt > 3) {
            // End reconnecting with built in error
            return undefined;
        }

        sails.log.debug('Handling redis retry strategy', {
            error: retryOptions.error,
            totalRetryTime: retryOptions.total_retry_time,
            attempt: retryOptions.attempt
        });

        // Reconnect after
        return Math.min(retryOptions.attempt * 100, 3000);
    }
});

Here is the stack trace of the error
AbortError: Redis connection lost and command aborted. It might have been processed.\n at RedisClient.flush_and_error (/var/sample-app/node_modules/redis/index.js:362:23)\n at RedisClient.connection_gone (/var/sample-app/node_modules/redis/index.js:664:14)\n at RedisClient.on_error (/var/sample-app/node_modules/redis/index.js:410:10)\n at Socket. (/var/sample-app/node_modules/redis/index.js:279:14)\n at emitOne (events.js:116:13)\n at Socket.emit (events.js:211:7)\n at emitErrorNT (internal/streams/destroy.js:66:8)\n at /var/sample-app/node_modules/async-listener/glue.js:188:31\n at args.(anonymous function) (/usr/lib/node_modules/pm2/node_modules/event-loop-inspector/index.js:138:29)\n at _combinedTickCallback (internal/process/next_tick.js:139:11)\n at process._tickDomainCallback (internal/process/next_tick.js:219:9)\n at process.fallback (/var/sample-app/node_modules/async-listener/index.js:563:15).

Our node app is running on a Docker container and it will connect to Redis server installed on an EC2 linux instance.

[mallikharjuna-ibexlabs]

We also facing same issue, but here we placed the stunnel to proxy the redis to localhost using below stunnel configuration.

fips = no
#setuid = root
#setgid = root
pid = /var/run/stunnel.pid
debug = 7
#options = NO_SSLv2
options = NO_SSLv3
output = /var/log/stunnel.log
#client = no
#foreground = yes

[redis-cli]
client = yes
accept = 127.0.0.1:6379
connect = redisendpoint:6379

node is disconnecting with these below errors for every couple of minutes. We running these through docker containers.

Redis connection to 127.0.0.1:6379 failed - read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
Error: Redis connection to 127.0.0.1:6379 failed - read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
Error: Redis connection to 127.0.0.1:6379 failed - read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
AbortError: Redis connection lost and command aborted. It might have been processed.

any resolution ?

[emps]

We also facing same issue, but here we placed the stunnel to proxy the redis to localhost using below stunnel configuration.

fips = no
#setuid = root
#setgid = root
pid = /var/run/stunnel.pid
debug = 7
#options = NO_SSLv2
options = NO_SSLv3
output = /var/log/stunnel.log
#client = no
#foreground = yes

[redis-cli]
client = yes
accept = 127.0.0.1:6379
connect = redisendpoint:6379

node is disconnecting with these below errors for every couple of minutes. We running these through docker containers.

Redis connection to 127.0.0.1:6379 failed - read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
Error: Redis connection to 127.0.0.1:6379 failed - read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
Error: Redis connection to 127.0.0.1:6379 failed - read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:111:27)
AbortError: Redis connection lost and command aborted. It might have been processed.

any resolution ?

same to me i think this is an issue with stunnel

[sudhanshugaur4]

I am also facing same error

{ Error: Redis connection to xx.xx.x.x:6379 failed - read ECONNRESET" 

at TCP.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' 

I am also using error handler but the thing is why is this happening, it can impact my query calls and delay my requests.

@ayaz Can you please open this issue because this issue is not completely resolved people only have handled error response but the problem is still unresolved.

[rujorgensen]

Bumping: Same issue here:

error: Error: Redis connection to xx.yy.zz.aa:6379 failed - read ECONNRESET at TCP.onStreamRead (internal/stream_base_commons.js:205:27) { errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' },

[robertop87]

I see the same in issue with:

• NodeJS v12.16.3
• NestJS framework
• IOredis: 4.2
• nestjs-redis: 1.2.8

Redis is provided by using Google Cloud MemoryStore.

The logs shows:

[ioredis] Unhandled error event: Error: read ECONNRESET

This is happening around one time each 24 hours without any specific reason or same steps, it just happen

[stevendinhdev]

@robertop87
I see the same issue. Redis is provided by using Google Cloud MemoryStore. Did you resolve this issue?

[hkd987]

Also seeing very similar issue with Cloud run / Cloud Functions when connecting to Cloud Memory Store. It possibly could be happening when the underlying resources are stopped too soon randomly by Google.

This issue just started about a week ago for us, with no changes to how we are handling the Redis connection. We are also using ioredis to handle the connection and running node 12.

[robertop87]

IORedis has as default strategy reconnect on error, but some reason it's not happening.

What I did is add the next in configuration:

reconnectOnError: (error): boolean => {
  console.log('Always reconnect on error', error)
  return true
}

The ECONNRESET error is still present, but now my connection is retried and my Google Instance is not being stuck anymore.

CC: @hkd987

Read more details about reconnectOnError here: https://github.com/luin/ioredis

[hkd987]

@robertop87 Made a new deployment with the specific configuration to always reconnectOnError set to return true. This seems like an issue that needs to be reopened or a new ticket made, since this is happening on the latest release.