bee-queue will not wait for redis re-connection and fails with error

[denigmus]

Testing bee-queue for a busy application. After connection with redis-server is lost, bee-queue will not wait for re-connection, but will fail. Sure, re-connection is a task of a redis-client, but seems that bee can't process it correctly when client reports disconnection. I'm surprised, is that as per design or a bug? I didn't find any setting for re-connection attempts or/and interval for that.

A alertsQueue error happened: NOSCRIPT No matching script. Please use EVAL. object { ReplyError: NOSCRIPT No matching script. Please use EVAL.
    at new Command (path\node_modules\redis\lib\command.js:12:22)
    at RedisClient.evalsha (path\node_modules\redis\lib\commands.js:58:47)
    at _commandable.then (path\node_modules\bee-queue\lib\queue.js:768:22)
    at <anonymous>
  command: 'EVALSHA',
  args:
   [ 'c61027c801f252df3cd499580cb716ca02e4bd1e',
     4,
     'bee:alertsQueue:stallBlock',
     'bee:alertsQueue:stalling',
     'bee:alertsQueue:waiting',
     'bee:alertsQueue:active',
     5000 ],
  code: 'NOSCRIPT' }
(node:20216) UnhandledPromiseRejectionWarning: ReplyError: NOSCRIPT No matching script. Please use EVAL.
    at new Command (path\node_modules\redis\lib\command.js:12:22)
    at RedisClient.evalsha (path\node_modules\redis\lib\commands.js:58:47)
    at _commandable.then (path\node_modules\bee-queue\lib\queue.js:768:22)
    at <anonymous>
(node:20216) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled
with .catch(). (rejection id: 1)
(node:20216) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
{ ReplyError: NOSCRIPT No matching script. Please use EVAL.
    at new Command (path\node_modules\redis\lib\command.js:12:22)
    at RedisClient.evalsha (path\node_modules\redis\lib\commands.js:58:47)
    at _commandable.then (path\node_modules\bee-queue\lib\queue.js:768:22)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
From previous event:

[denigmus]

I thought ioredis will improve something, but both bee-queue: 1.x and bee-queue 2.0dev behave the same on redis stop, here is from 2.0dev:

A alertsQueue error happened: connect ECONNREFUSED 127.0.0.1:6379 object { Error: connect ECONNREFUSED 127.0.0.1:6379
    at Object._errnoException (util.js:992:11)
    at _exceptionWithHostPort (util.js:1014:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1186:14)
  code: 'ECONNREFUSED',
  errno: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 6379 }
(node:14928) UnhandledPromiseRejectionWarning: ReplyError: NOSCRIPT No matching script. Please use EVAL.
    at parseError (pathhere\node_modules\ioredis\node_modules\redis-parser\lib\parser.js:179:12)
    at parseType (pathhere\node_modules\ioredis\node_modules\redis-parser\lib\parser.js:302:14)
(node:14928) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled
with .catch(). (rejection id: 1)
(node:14928) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

After redis start:

{ ReplyError: NOSCRIPT No matching script. Please use EVAL.
    at parseError (pathhere\node_modules\ioredis\node_modules\redis-parser\lib\parser.js:179:12)
    at parseType (pathhere\node_modules\ioredis\node_modules\redis-parser\lib\parser.js:302:14)
From previous event:
   
......

    at Socket.ondata (_stream_readable.js:639:20)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
    at readableAddChunk (_stream_readable.js:250:11)
    at Socket.Readable.push (_stream_readable.js:208:10)
    at TCP.onread (net.js:597:20)
  command:
   { name: 'evalsha',
     args:
      [ '6ecea656d685389e1ac506df66563aa29424c126',
        '3',
        'bee:alertsQueue:id',
        'bee:alertsQueue:jobs',
        'bee:alerts
.....

[harryhazza77]

i'm seeing the same problem i.e. after a redis restart, i get

ReplyError: NOSCRIPT No matching script. Please use EVAL.
  at parseError (/usr/src/app/node_modules/redis-parser/lib/parser.js:193:12)
  at parseType (/usr/src/app/node_modules/redis-parser/lib/parser.js:303:14)

[frank-dspeed]

i think you need to code such logic your self

[DanielTamkin]

I'm getting similar errors. During transit, if i end a redis-server bee-queue throws a hard crash with no obvious methods to mitigate this.

[Tob0t]

We are having the same problem, if the redis connection is lost for some seconds, bee-queue is not able to create jobs anymore, any ideas how to handle this?

[frank-dspeed]

@Tob0t deploy a redis cluster

[DanielTamkin]

This is a less than helpful response @frank-dspeed. We're asking for clarity on how to handle disconnects from between redis & bee-queue. Not mitigate by supplying more endpoints for bee-queue to connect to.
Until this is reconized, @Tob0t i found a solution. Create a redis client separate from bee-queue(npm --save redis). This'll allow you to develop retry strategies* / handle disconnects emitted from the client.

• Specifically the function retry_strategy was of major use to attempt a reconnect.

[Tob0t]

Thanks @DanielTamkin I solved it now by catching the error on create job and then reloading the scripts as suggested by @skeggse (#119 (comment)). It's maybe not working for every use case but it fits for ours.

import bqScripts from 'bee-queue/lib/lua';

queue.createJob(config)
      .save(async (err, job) => {
        if (err) {
          console.error(`failed creating job ${id}`);
          // Known error when redis has not all lua scripts loaded properly
          if (err.command === 'EVALSHA') {
           await bqScripts.buildCache(redisClient);
           console.info('Successfully reloaded Lua scripts into cache!');
           // create job again
           queue.createJob(config).save();
         }
       }
     });

[shils]

@skeggse any reason not to catch NOSCRIPT No matching script errors in Queue#_evalScript, build the script cache, and retry?

[pskopnik]

From my point of view this issue is quite significant, I've stumbled upon this after only a few hours of testing. For production use, resilience to transient outages is paramount.

The recommendation by Redis is exactly @shils suggestion: Optimistically perform an EVALSHA and if the NOSCRIPT error is returned perform an EVAL. Some redis clients include functionality for doing exactly this (though the redis package does not).

#316 looks promising in this regard.

Preloading scripts as triggered by Queue#ready() does not interfere with this approach.