Need help in finding a better solution

[wernermorgenstern]

We have the following use case:
We have devices, which send a notification every 10 seconds to the MQTT Broker (VerneMQ).
We have a client service, which connects to the MQTT Broker and subscribes to the Topic, and then handles the incoming messages. The QoS for the subscription is 2.

The client service runs on AWS, in Kubernetes (EKS)

Each device sends a message every 10 seconds.

When we have a few thousand (like around 6884 right now) of these devices, we can't keep up on the Client Side.
For 6884 devices, at every 10s per device, we are getting 41304 messages per minute.

I have currently running 10 instances of the Client Service, but we are still behind by 2 minutes or more.
Each message gets processed, and when I put timing logging statements, each messages takes about 10 milliseconds.

I have seen that MQTT.js gets multiple messages in each connection (so it seems to be batching them).

What other ways can I implement it in NodeJS? (We use Typescript, and convert to Javascript, and then run the Javascript code).

I was thinking Worker Threads, but before I go and put the effort in, I want to actually make sure that would help in solving the issue.

I found two libraries, which implement Worker Threads with easier/simpler ways.
https://www.npmjs.com/package/piscina
https://github.com/poolifier/poolifier

I don't necessarily want to increase instances. We want to put up the Devices to 10_000, and even 50_000, and even higher.

[robertsLando]

Dup of #1692 please don't open new issues with the same topic multiple time, comment on the other one if you want to add context

[wernermorgenstern]

@robertsLando , the other one has been closed already. This is a new issue. It is the same use case, but now I don't have the issue with the disconnecting anymore. Now the issue is that the client can't keep up anymore

[robertsLando]

I answered you there as them are strictly related