Client connected to the two nodes at the same time

[Nowogrodzki]

Hello, yesterday I discovered a strange situation, namely several clients have been connected two both nodes(while investigating the status in is_online shows false).
Also, our plugin for detecting whether a client is online or not began to act strangely(as shown in the screenshot)

Chronologically it looks like this

• Client connected to the vernemq-1
• Client can't register to the vernemq-0 due to action_timeout
• Client connected to the vernemq-0
• Client disconnected from the vernemq-0
• Client disconnected from the vernemq-1
• Client connected to the vernemq-1
• Client can't register to the vernemq-1 due to action_timeout
• Client can't register to the vernemq-0 due to action_timeout
• Client connected to the vernemq-0
• Client disconnected to the vernemq-0
• Client connected to the vernemq-0
• Client disconnected to the vernemq-1
  We ended up in a situation where our plugin sent an event that the client is offline from vernemq-1 at the time it was connected to the vernemq-0

My expectation -

• Client is disconnected from the node x before he's connected to node y
• Client should be only visible at one node(no duplicate among nodes)

Plugin code

Setup on Kubernetes -
Two nodes of vernemq deployed from statefulset

Variables configured in the vernemq.conf -

allow_register_during_netsplit = off
allow_publish_during_netsplit = off
allow_subscribe_during_netsplit = off
allow_unsubscribe_during_netsplit = off
allow_multiple_sessions = off
coordinate_registrations = on
persistent_client_expiration = 3m
max_online_messages = 100000
max_offline_messages = 100000
upgrade_outgoing_qos = on
max_message_size = 0
listener.max_connections = infinity
listener.nr_of_acceptors = 100
listener.tcp.allowed_protocol_versions = 3,4,5,131
shared_subscription_policy = prefer_online_before_local
receive_max_client = 1000
topic_max_depth = 20

[ioolkos]

@Nowogrodzki

For clarity on theEULA-based binary packages: you need a paid subscription to use them commercially.

Can you see more in the logs on why the action_timeout is happening? I can imagine that you use persistent sessions and trigger queue migration. You might want to address this by using a "sticky session" load balancing strategy.
Do you see this happening for a specific client ID? if yes, is there anything specific to the behaviour of this client?

[Nowogrodzki]

Ahh sorry I forget to mention that we are using vernemq by creating our own distribution

Sadly this kind of issue happened to several clients in production, regarding load-balancing as far as I know, our strategy is set to high availability. Tomorrow when our DevOps be back we will try to change it to sticky.

Currently, I don't really know what action_timeout happened but I'll investigate it and will update here.

P.S
Yes we are using persistent sessions with QoS 1

[rafsawicki]

We've written the plugin (I work with @Nowogrodzki ) with the assumption that a client with the same client id will never be connected to the cluster twice, and based on that we publish an online status on MQTT for that client in on_client_wakeup, on_client_offline and on_client_gone hooks.

• Was this assumption incorrect? We're not sure if what we're seeing is a bug, or if VerneMQ simply doesn't provide this guarantee of a single connection in clustering mode.
• Is there a better (built in?) way to keep the backend application updated about the client connection status, that would correctly propagate that status in the multiple connections scenario we're facing?

[ioolkos]

@rafsawicki Hi. You're not seeing a bug, you're seeing effects.
They come from 2 areas:

• sync of multiple registration attempts. VerneMQ makes sure that only 1 registration attempt is handled when concurrent attempts happen for the same ClientID. That's why you see action_timeout. (Queue migration also plays a role here, as you might be in a situation where Verne just cleans up behind your register attempts).
• All notification hooks are just that: notifications of an event. For most of the time, you will benefit from the "correct order" of events, if you use them to build a presence service. For a low number, you will see concurrent interleavings (clients connects on 1 node, kicks itself off from another node, but the on_client_gone happens latest making your presence service think that your client is offline).

You can probably address this by some added heuristics, timestamps, by sending birth and death messages to a service, or even by additional calls to the new v2 API: #2132

Note that Clients should properly disconnect, not just throw new instances of the same ClientId at the broker as a systematic method to re-connect. This will likely also help with linearity.

Apologies for not writing up a more extended answer.

[rafsawicki]

Thank you for all of the suggestions, providing more context, and helping us to better understand notification hooks and the VerneMQ behavior during the client connection. Unfortunately, we're not able to fully control MQTT clients, since they are provided by a different company, so we have to work around some of their limitations, like not properly disconnecting in some instances. Since we weren't able to easily adjust the load balancing strategy, as our load balancers do not support TLS alongside sticky sessions, we're going to adjust our presence service and add a heuristic to keep track of the last node that the client connected to, discarding disconnect messages coming from different nodes.