Webhook loadbalancing does not work from VerneMQ

[henningw]

Environment

• VerneMQ Version: 1.12.5

• OS: CentOS (VerneMQ in docker debian 11)

• Erlang/OTP version (if building from source): 24

• VerneMQ configuration (vernemq.conf):
  vmq_webhooks.mywebhook11.hook=auth_on_register
  vmq_webhooks.mywebhook11.endpoint=http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/register
  vmq_webhooks.mywebhook12.hook=auth_on_subscribe
  vmq_webhooks.mywebhook12.endpoint=http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/subscribe
  vmq_webhooks.mywebhook13.hook=auth_on_publish
  vmq_webhooks.mywebhook13.endpoint=http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/publish
  vmq_webhooks.mywebhook13.no_payload=on
  vmq_webhooks.mywebhook21.hook=auth_on_register
  vmq_webhooks.mywebhook21.endpoint=http://10.128.0.2:8090/XXX/api/auth/v1/vernemq/register
  vmq_webhooks.mywebhook22.hook=auth_on_subscribe
  vmq_webhooks.mywebhook22.endpoint=http://10.128.0.2:8090/XXX/api/auth/v1/vernemq/subscribe
  vmq_webhooks.mywebhook23.hook=auth_on_publish
  vmq_webhooks.mywebhook23.endpoint=http://10.128.0.2:8090/XXX/api/auth/v1/vernemq/publish
  vmq_webhooks.mywebhook23.no_payload=on

• Cluster size/standalone:
  standalone

Expected behaviour

VerneMQ should balance the webhook connections to two HTTP endpoints.

Actual behaviour

VerneMQ is only sending requests to the first configured webhook server

Output from vmq-admin

# docker exec -it XXX-cloud-vernemq-service vmq-admin webhooks show
+-------------------+------------------------------------------------------------+---------------+------------------+
| hook              | endpoint                                                   | base64payload | response_timeout |
+-------------------+------------------------------------------------------------+---------------+------------------+
| auth_on_subscribe | http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/subscribe | true          | 5000             |
+-------------------+------------------------------------------------------------+---------------+------------------+
| auth_on_subscribe | http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/subscribe | true          | 5000             |
+-------------------+------------------------------------------------------------+---------------+------------------+
| auth_on_register  | http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/register  | true          | 5000             |
+-------------------+------------------------------------------------------------+---------------+------------------+
| auth_on_register  | http://10.128.0.2:8090/XXX/api/auth/v1/vernemq/register  | true          | 5000             |
+-------------------+------------------------------------------------------------+---------------+------------------+
| auth_on_publish   | http://10.128.0.2:8090/XXX/api/auth/v1/vernemq/publish   | true          | 5000             |
+-------------------+------------------------------------------------------------+---------------+------------------+
| auth_on_publish   | http://10.128.0.2:8090/XXX/api/auth/v1/vernemq/publish   | true          | 5000             |
+-------------------+------------------------------------------------------------+---------------+------------------+

# docker exec -it XXX-cloud-vernemq-service vmq-admin webhooks cache show
+---------+------------------------------------------------------------+-------------------+-------+
| stat    | endpoint                                                   | hook              | value |
+---------+------------------------------------------------------------+-------------------+-------+
| misses  | http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/subscribe | auth_on_subscribe | 17922 |
+---------+------------------------------------------------------------+-------------------+-------+
| misses  | http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/register  | auth_on_register  | 8824  |
+---------+------------------------------------------------------------+-------------------+-------+
| misses  | http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/publish   | auth_on_publish   | 738   |
+---------+------------------------------------------------------------+-------------------+-------+
| hits    | http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/subscribe | auth_on_subscribe | 2757  |
+---------+------------------------------------------------------------+-------------------+-------+
| hits    | http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/register  | auth_on_register  | 11425 |
+---------+------------------------------------------------------------+-------------------+-------+
| hits    | http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/publish   | auth_on_publish   | 350   |
+---------+------------------------------------------------------------+-------------------+-------+
| entries | http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/subscribe | auth_on_subscribe | 402   |
+---------+------------------------------------------------------------+-------------------+-------+
| entries | http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/register  | auth_on_register  | 1416  |
+---------+------------------------------------------------------------+-------------------+-------+
| entries | http://10.128.0.1:8090/XXX/api/auth/v1/vernemq/publish   | auth_on_publish   | 118   |
+---------+------------------------------------------------------------+-------------------+-------+

[ioolkos]

@henningw thanks, yes, that's not implemented.
We'd have to check whether Hackney (HTTP client) supports connection pooling to 2 endpoints.

Is the reason you tried to set this up actual performance issues with 1 endpoint? (it mostly matters for MQTT registers, not publish and subscribes where the authorization calls hit the WebHooks cache).

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.

[henningw]

Thanks for the reply. Would be helpful if it would be documented e.g. in here or in the cfg.

Regarding performance, the problem is that negative authentication results are not cached (#1242), and some devices are buggy regarding subscription behaviour.

[ioolkos]

@henningw See your point. The thing is Verne needs to hit the authentication endpoint really for every single register, as it's the source of truth here. Caching seems problematic.
We could load balance, somehow adding a lane to a traffic problem. I guess ideally we'd had some way of giving back a throttling modifier as in the auth_on_publish hook. Unfortunately that's the only hook where throttling is currently possible.

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.