[Bug]: CRASH REPORT Process with 0 neighbours crashed with reason: bad argument in vmq_cluster_com:process_bytes/3 line 171

[ashtonian]

Environment

• VerneMQ Version: 1.12
• OS: eks
• Erlang/OTP version (if building from source): 25.0
• Cluster size/standalone: 3

Current Behavior

Cluster seems unstable and seems to be unable to sync.

14:33:36.287 [warning] Proxy Protocol Error: 'The PROXY protocol header signature was not recognized. (PP 2.1, PP 2.2)'
14:33:36.737 [info] Replica meta5: AE exchange with 'VerneMQ@v-1.v-headless.lab.svc.cluster.local' synced 543 objects
14:33:37.241 [info] Replica meta6: AE exchange with 'VerneMQ@v-1.v-headless.lab.svc.cluster.local' synced 566 objects
14:33:38.222 [info] Replica meta9: AE exchange with 'VerneMQ@v-0.v-headless.lab.svc.cluster.local' synced 112 objects
14:33:38.577 [info] Replica meta2: AE exchange with 'VerneMQ@v-1.v-headless.lab.svc.cluster.local' synced 566 objects
14:33:39.205 [info] Replica meta3: AE exchange with 'VerneMQ@v-0.v-headless.lab.svc.cluster.local' synced 127 objects
14:33:40.469 [info] Replica meta7: AE exchange with 'VerneMQ@v-0.v-headless.lab.svc.cluster.local' synced 99 objects
14:33:45.122 [info] Replica meta8: AE exchange with 'VerneMQ@v-1.v-headless.lab.svc.cluster.local' synced 468 objects
14:33:45.148 [info] Replica meta1: AE exchange with 'VerneMQ@v-1.v-headless.lab.svc.cluster.local' synced 450 objects
14:33:45.884 [info] Replica meta3: AE exchange with 'VerneMQ@v-0.v-headless.lab.svc.cluster.local' synced 127 objects
14:33:54.130 [warning] session stopped abnormally due to '{cant_parse_connect_fixed_header,<<"GET /metrics HTTP/1.1\r\nHost: 10.10.36.109:1884\r\nUser-Agent: Prometheus/2.49.1\r\nAccept: application/openmetrics-text;version=1.0.0;q=0.5,application/openmetrics-text;version=0.0.1;q=0.4,text/plain;version=0.0.4;q=0.3,*/*;q=0.2\r\nAccept-Encoding: gzip\r\nX-Prometheus-Scrape-Timeout-Seconds: 10\r\n\r\n">>}'
14:33:55.873 [info] Replica meta5: AE exchange with 'VerneMQ@v-1.v-headless.lab.svc.cluster.local' synced 543 objects
14:33:56.564 [info] Replica meta2: AE exchange with 'VerneMQ@v-1.v-headless.lab.svc.cluster.local' synced 566 objects
14:33:57.000 [info] Replica meta6: AE exchange with 'VerneMQ@v-1.v-headless.lab.svc.cluster.local' synced 566 objects
14:33:59.618 [info] Replica meta6: AE exchange with 'VerneMQ@v-0.v-headless.lab.svc.cluster.local' synced 106 objects
14:33:59.751 [info] Replica meta7: AE exchange with 'VerneMQ@v-1.v-headless.lab.svc.cluster.local' synced 515 objects
14:33:59.809 [error] CRASH REPORT Process <0.4376.142> with 0 neighbours crashed with reason: bad argument in vmq_cluster_com:process_bytes/3 line 171

Expected behaviour

No crash report

Configuration, logs, error output, etc.

No response

Code of Conduct

• I agree to follow the VerneMQ's Code of Conduct

[ioolkos]

@ashtonian Thanks. I can't remember seing that last error line, will investigate.
Clearly, your cluster is out of sync since it does empty synchronization (certain batches, like 566, are always repeating).
States like that are introduced by cluster joins with non-empty histories. It's not clear what exact version you use; newer releases have mitigated some of the effects. The latest release has a configurable setting to completely prevent non-empty SWC nodes to join a cluster.
The only quick and clear procedure to resolve this is unfortunately a full reset of the cluster, ie. scaling down to 0 and up again. (be aware this might loose state in the cluster).

---

👉 Thank you for supporting VerneMQ: https://github.com/sponsors/vernemq
👉 Using the binary VerneMQ packages commercially (.deb/.rpm/Docker) requires a paid subscription.