redis-smq has better performance than bull, what to choose?

[AmreeshTyagi]

@weyoss Thanks for nicely crafted library.

I have done some performance benchmark to compare redis-smq with bull on my Dell XPS. Machine 16GB RAM, Hexa Core

No doubt, redis-smq is clear winner because of it's simplicity and focused approach towards message queuing.

Following is the messages produced and consumed rate

Scenarios	redis-smq produce rate and consumed rate	bull produce rate and consumed rate
1 producer and 1 consumer	16.5k/sec and 9k/sec	9.5k/sec and 7k/sec
1 producer and 2 consumer	15k/sec and 13.5k/sec	5.0k/sec and 5.0k/sec
2 producer and 2 consumer	23.3k/sec and 13.2k/sec	13.3k/sec and 10.6k/sec
4 producer and 4 consumer	30k/sec and 17.8k/sec	13.4k/sec and 10.4k/sec
6 producer and 3 consumer	40k/sec and 16.1k/sec	17.5k/sec and 12.2k/sec

It is clearly visible that bull is almost saturated in terms of producer and consumer rate.
There can be multiple reasons of less performance in bull like promises or multiple checks & conditions to provide other features like rate limiting etc.

Though, I am not able to convince myself which library to use for my next project, because redis-smq simplicity and performance is far ahead of bull, which is the purpose of this lib, but bull has lot of features, which can be useful when things are in production.

I am thinking to move from RabbitMQ to Redis, because I don't need complex routing like RabbitMQ.
I have also seen heavy usage for simple queuing task in RabbitMQ, where RabbitMQ was taking approx 2GB for holding 1 million messages and total messages size was around 100MB. So ideally in Redis, it would be 100MB only. I understand RabbitMQ do lot of things out of the box, but I think Redis can work very well in simple queuing cases.

Any suggestion would be appreciated by anyone to take this decision. I am also interested to listen more from actual user of this library who have used it in production with heavy load.

Thanks.

[weyoss]

@AmreeshTyagi Thank you for considering this library for your future project and for sharing your benchmarks. This library is indeed focused on performance and simplicity. My advise to you, although it may not be in time, is to try it. Also you should consider the capabilities and the features the library provides. It is not a drop-in replacement for RabbitMQ in terms of features. I believe queuing a message and delivering it later to a consumer should be a simple task with minimal resource requirements and without introducing complex layers to your application. That is the main reason why this library was created. If you have any issue or a question, I will be glad to help you.

[wackfx]

Throwing my 2cts here; I'm writing this as I've been dealing with both bull and smq - and we don't often celebrate great work.

Bull never really worked (at least for me) - I have an application that deals with ~10k messages/sec and I had a lot of memory issues with it (my consumer / producers would stop working after a couple of hours - with memory going up without apparent reason).

As I used an onion architecture (fortunately), I was able to switch from bull to redis-smq in an afternoon and all the issues were solved.

Only issue I had with redis-smq was the fact that version 7 didn't support point-2-point, but now it does. Migrating to v8 now and so far no issues, migration has been silky smooth.

Anyway, can't recommend redis-smq enough, it's probably the best piece of software to manage microservices with node (and bun also!). Very great work @weyoss !

EDIT: While i'm at it, a bit off-topic (but not worth the noise of an issue IMO) - I would love to have a documentation on v8 about cleanup (Best practice about closing producer / queue / consumer ? which functions ? should I .cancel() consumer before actually closing it ?)

[weyoss]

@wackfx It's my pleasure!

Thank you for using redis-smq and also for sharing your delightful experience with the message queue. Your feedback is highly appreciated.

Sure. I will try to clarify the points you highlighted and in general the cleanup process and include it into the documentation.