Message throughput is less than expected, looking for troubleshooting suggestions

[rshillington]

I have a set of 20,000 messages (message size ~2K) and with QoS = 1 it takes about 6 minutes to publish to a 2 node cluster.
That's about 50 messages per second.
If I publish the same set of messages with QoS=0 it takes about 9 seconds to publish all the messages. I think there must be something else going on to account for that difference in throughput.

[ieQu1]

Hello.

We need more details to start looking into it. For starters:

• How many clients publish the messages?
• What is Receive-Maximum on the clients?
• How the parameters listed here are configured? https://www.emqx.io/docs/en/latest/design/inflight-window-and-message-queue.html#inflight-queue-and-receive-maximum
• How many subscribers is there?
• Do you have any rule engine rules configured?
• What is the CPU load on the broker nodes?

Some leads: if CPU load is NOT high, and there are no overload alarms, then the prime suspect is the flow control (receive-maximum and the inflight window settings). With QoS1, both client and the broker limit the number of inflight messages existing in the network. As a result, the network latency starts to affect the throughput, as both client and the broker have to pause the transmission until there is space in the inflight buffer. This can be solved by increasing the inflight size or paralellizing the transfers between multiple clients.

[rshillington]

I'm trying to do this is a controlled environment:

• There is only one client publishing messages
• I'm using using mqttx pub cli.
  • I was not setting the --receive-maximum -- so I tried again with a setting of 10000 which didn't make a difference so I suspect I'll ahve to change something in the configuration of the broker?
• There are no subscribers to this particular topic
• There is no rules configured
• the CPU load at the broker 0.01/0.03/0

The broker is running in AWS and the client is running on a macOS machine with a 1Gb internet connection so I'm doubtful that network latency is an issue.

I also checked the listener management page on the console and nothing is set:

[ieQu1]

with a 1Gb internet connection so I'm doubtful that network latency is an issue.

1Gb/s is a measure of bandwidth, while I am talking about latency (measured in seconds). By default, max_inflight for the external listener is just 32 (this setting is optimized for handling multitude of low-traffic clients, rather than a single high-traffic one): https://www.emqx.io/docs/en/latest/design/inflight-window-and-message-queue.html#configuration-items
If it takes, say, 10 ms for the PUBLISH packet to reach the broker, and 10 ms for the PUBACK to reach the client, and only 32 packets unacked packets can exist in connection at a time, then most of the time the connection is idle.
Can you try to increase this setting?

[rshillington]

Unless I'm missing something about configuration of the broker (which is quite possible) I'm not seeing any difference:
This it the current emqx.conf file

node {
  name = "emqx@127.0.0.1"
  cookie = "emqxsecretcookie"
  data_dir = "/var/lib/emqx"
}

cluster {
  name = emqxcl
  discovery_strategy = manual
}

dashboard {
    listeners.http {
        bind = 18083
    }
}

session {
    max_subscriptions = infinity
    upgrade_qos = false
    max_inflight = 65535
    retry_interval = 30s
    max_awaiting_rel = 100
    await_rel_timeout = 300s
    session_expiry_interval = 2h
    max_mqueue_len = 1000
    mqueue_priorities = disabled
    mqueue_default_priority = lowest
    mqueue_store_qos0 = true
    
    force_shutdown {
      max_message_queue_len = 1000
      max_heap_size = 32MB
    }

    force_gc {
      count  =  16000
      bytes  =  16MB
    }
  }

[ieQu1]

Your configuration doesn't pass check on 5.6:

2024-05-07T09:32:45.798841+02:00 [warning] msg: config_key_not_recognized, unknown_config_keys: session
You probably meant mqtt. force_gc and force_shutdown are root subtrees, etc.