VerneMQ connections not closing causing it to crash

[mbitar-gm]

Environment

• VerneMQ Version: 1.11.0
• VerneMQ configuration (vernemq.conf) or the changes from the default:

listener.http.metrics=0.0.0.0:8888
max_offline_messages=30000
max_online_messages=30000
listener.max_connections=25000

Actual behaviour

I have a VerneMQ broker running with metrics enabled, and I visualize these on a Grafana dashboard.
I have over 5200 devices connected to that broker. These devices restart their connections every approximately 16 hours.

Usually everything goes smoothly, devices restart their connections, they reconnect normally, and no issues seem to happen with the broker.

However, sometimes around restarts, the following happens:

• Socket Open almost becomes the double of Socket Close:
  

• Consequently the number of connected clients keeps going up until broker stops functioning properly:
  

Around that time I can observe the following errors in the logs :

"22:46:24.232 [error] CRASH REPORT Process <0.3534.183> with 0 neighbours crashed with reason: no try clause matching {shutdown,{state,<<>>,vmq_mqtt_pre_init,terminated,mqttws,#Port<0.276347>,{{172,18,0,5},54939},{{1663,281924,69525},0},{{1663,281924,69525},0}}} in cowboy_websocket:handler_call/6 line 469\r"
"22:46:24.233 [error] Ranch listener {{0,0,0,0},8080} terminated with reason: no try clause matching {shutdown,{state,<<>>,vmq_mqtt_pre_init,terminated,mqttws,#Port<0.276347>,{{172,18,0,5},54939},{{1663,281924,69525},0},{{1663,281924,69525},0}}} in cowboy_websocket:handler_call/6 line 469\r"
"22:46:24.280 [warning] Subquery failed due to timeout\r"

After that happens, all clients are unable to connect to the broker.
Unfortunately I was not able to reproduce the issue yet as it happens haphazardly.

Update:

Also found out the following happens after the issue starts happening:

• Messages get queued up:
  

• Commands that would usually take 1 second to return start timing out after 1 minute.

[ioolkos]

Check what the upstream network components do. Do you have any idle timeouts configured in a firewall?
Keep us posted on your findings.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[mbitar-gm]

Check what the upstream network components do. Do you have any idle timeouts configured in a firewall? Keep us posted on your findings.

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq 👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

I don't think this is a Network issue.
We have a cronjob that queries the broker once an hour, and when the issue starts commands start timing out.

Updated the post.

[mbitar-gm]

Bump

[ioolkos]

Does this look like a reconnect overload? Do you actually see CPU maxed, maybe running on a supertiny instance?
How long do the clients wait before they get a connack answer (and before they would do another immediate connect attempt).

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.