How about consider use Redis stream structure to implement MQ

[cw1427]

It is not a issue, I just want to discuss about Redis stream consumer group to implement MQ would be more easier for the tasks queue implement. How about your opinion?

[selwin]

I agree with you. I think Redis stream would be perfect for RQ. This is something we can consider for the future, probably a few years in when more systems ships with Redis 5.

[cw1427]

@selwin : Do we have plan to implement it? Cause Redis 6 GA is published, I'm so eager to have Steam feature to make RQ be more reliable and flexible.

[selwin]

@cw1427 I think it's still a bit early to move to Redis streams because most server distros still ship with Redis 4.X by default.

You mentioned that adopting Streams will make RQ more reliable and flexible. I'm just curious as to what features you want to see in RQ and whether we can implement them without switching to Redis Streams.

As for myself, the most prominent benefit I see in switching to Streams would be the capability to XACK jobs so they're never dropped in cases where hard failures happen after popping the job from the queue.

[cw1427]

@selwin Yes the ack would be the most charming feature to make it be more reliable, and also the consumer group feature that Redis Stream has to make it flexible to deal with mulity kinds messages generation by different applications like "my Gerrit hook events, my Jenkins jobs trigger event, my JIRA actions and so on".

I would prefer RQ could be the general "pipe" like event bus to keep all of messages coming like steam and be consumed by customer clients by goups.

Here I fould a project shipped to Redis stream already. https://github.com/robinjoseph08/redisqueue But it is Go stack. I'm not good at customizing it than RQ.

[nkumar15]

redis stream is also useful for autoscaling cases in k8s cluster. Keda supports redis stream, if RQ supports redis streams it will be a neat and complete solution for a job queue based autoscaling functionality on K8s systems.

[selwin]

This commit implements RQ's first use of Redis Stream to store data. What kind of use cases are we talking about here? If we were to implement or move something to Redis Stream, there has to be an upside.

[nkumar15]

thanks @selwin for quick response. The use case which I am seeing currently is to increase or decrease rq worker counts which are running in containers on basis of how many messages are present in redis queue.

The underlying infrastructure is Kubernetes for it and autoscaling is supported by KEDA

KEDA can monitor redis streams and increase underlying container counts as per set threshold.

If RQ supports redis streams then autoscaling of RQ workers can be easily done through KEDA

[selwin]

Can’t Keda do this by looking at number of elements in a Redis list?

…

On Nov 29, 2022 at 16:30:20, Neeraj Kumar ***@***.***> wrote: thanks @selwin <https://github.com/selwin> for quick response. The use case which I am seeing currently is to increase or decrease rq worker counts which are running in containers on basis of how many messages are present in redis queue. The underlying infrastructure is Kubernetes for it and autoscaling is supported by KEDA <https://keda.sh/> KEDA can monitor redis streams <https://keda.sh/docs/2.8/scalers/redis-streams/>and increase underlying container counts as per set threshold. If RQ supports redis streams then autoscaling of RQ workers can be easily done through KEDA — Reply to this email directly, view it on GitHub <#1144 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABOB4R7Y7WICMFS3W6PN6TWKXEKZANCNFSM4JAZMUHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[nkumar15]

yes KEDA supports Redis list also

[essamgouda97]

KEDA supports Redis lists but RQ uses redis sets which are not supported by KEDA.

Is that true or does rq use redis lists ? @selwin I tried this

127.0.0.1:6379> type rq:workers:TEST-Q
 set

[jheyer159]

Currently working on setting this up

KEDA supports Redis lists but RQ uses redis sets which are not supported by KEDA.

Is that true or does rq use redis lists ? @selwin I tried this

127.0.0.1:6379> type rq:workers:TEST-Q
 set

According to the following code pushing a job to the queue is using the Redis commands LPUSH and RPUSH

rq/rq/queue.py

Lines 451 to 454 in 60164c5

	if at_front:
	result = connection.lpush(self.key, job_id)
	else:
	result = connection.rpush(self.key, job_id)

which can be verified here as a list command.

EDIT 3/20/2023

after building out the PoC for this the main unknown was connecting the ScaledObject correctly which was discovered as follows.

triggers:
    - type: redis
      metadata:
        listName: rq:queue:my-queue-name   # <-- the queue name you're using - notice the prefix rq:queue

for example:

from rq import Queue
q = Queue('my-queue-name')