Streaming server response

[aletorrado]

Hi, is possible with node_redis to stream the server response with commands like HMGET or KEYS?

Thanks.

[hossam-fares]

👍 it would be great if we have implementation for it

[hossam-fares]

@aletorrado I already had similar situation and I wanted to stream from HKEYS cmd
so I created a readable stream using HSCAN cmd, with this implementation:

const redis = require('redis');
const connectionParams = {};

class HKeysReadable extends require('stream').Readable {
  constructor(key, chunkSize, opt) {
    opt = opt || {};
    opt.objectMode = true;
    super(opt);
    this._cursor = 0;
    this.key = key;
    this.chunkSize = chunkSize;
    this.client = redis.createClient(connectionParams);
  }

  _read() {
    if (this._cursor === '0') {
      this.push(null);
      return this.client.quit();
    }

    this.client.hscan(this.key, this._cursor, 'COUNT', this.chunkSize, (err, res) => {
        if (err) {
          this.push(null);
          this.client.quit();
          return process.nextTick(() => this.emit('error', err));
        }

        this._cursor = res[0];

        /**
         *
         SCAN returns value as an array of two values, the first value is the new cursor to use in the next call,
         the second value is an array of elements like: [key1, val1, key2, val2, key3, val3, .....]
         In this stream, we need the keys only so we reduce that array to get only the odd values which contain the keys

         */
        let keys = res[1].reduce((accumulator, val, index)=> {
          if (!(index % 2)) {
            accumulator.push(val);
          }

          return accumulator;
        }, []);

        if (keys.length > 0) {
          this.push(keys);
        }
      }
    );
  }
};

@BridgeAR what do you think? is there a possibility to adapt similar implementation in Redis library as well?

[aletorrado]

Great @hossam-fares. Basically it may abstract cursors from node_redis clients, for a whitelisted set of redis commands (ie. SCAN, SSCAN, HSCAN, ZSCAN).

[BridgeAR]

@hossam-fares I would definitely like to have something similar implemented 👍

[hossam-fares]

@BridgeAR cool, I will work on a PR for it

[revington]

Has this been implemented?

[leibale]

@aletorrado @hossam-fares @revington

I think that AsyncIterator is more suitable:

for await (const key of client.scanIterator()) {
  console.log(key);
}

what do you think?