Connection to HiveMQ dropping with an error

[nicholasmparker]

Anyone experiencing this? From the debug logs
2024-03-01 10:47:36.005 E MessageProcessorEndpointMqtt: connectionLost error
(32109) - java.io.EOFException
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(Unknown Source:213)
at java.lang.Thread.run(Thread.java:1012)
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:275)
at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(SourceFile:14)
at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(Unknown Source:64)
... 1 more

[jpmens]

Under the assumption that this error is on our Android app, I'll transfer this issue.

[growse]

Difficult to know what's going on here. Could be the network, could be the broker not being responsive, could be the device not waking the app to do keepalives....

[da-mkay]

Hi,

I have exactly the same issue:

2024-03-18 22:11:13.042 E MessageProcessorEndpointMqtt: connectionLost error
2024-03-18 22:11:13.042 E MessageProcessorEndpointMqtt: (32109) - java.io.EOFException
    at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(Unknown Source:213)
    at java.lang.Thread.run(Thread.java:1012)
2024-03-18 22:11:13.042 E MessageProcessorEndpointMqtt: Caused by: java.io.EOFException
    at java.io.DataInputStream.readByte(DataInputStream.java:275)
    at org.eclipse.paho.client.mqttv3.internal.wire.MqttInputStream.readMqttWireMessage(SourceFile:14)
    at org.eclipse.paho.client.mqttv3.internal.CommsReceiver.run(Unknown Source:64)
2024-03-18 22:11:13.042 E MessageProcessorEndpointMqtt: ... 1 more

The Android phone (Samsung Galaxy S21 FE) connects to the broker, keeps the connection for "exactly" 1 minute, then disconnects. Then I see the error above.
But then it immediately reconnects. Thus, in my broker logs I see an LWT message and reconnect every minute:

[2024-03-18 22:10:12.523] moe connected
[2024-03-18 22:10:12.540] moe subscribed topics: owntracks/+/+ owntracks/+/+/event owntracks/+/+/info owntracks/+/+/waypoints owntracks/moe/phone/cmd
[2024-03-18 22:11:12.585] moe published to owntracks/moe/phone :
[2024-03-18 22:11:12.585] {"_type":"lwt","tst":1710796212}
[2024-03-18 22:11:12.585] moe disconnected

[2024-03-18 22:11:13.454] moe connected
[2024-03-18 22:11:13.470] moe subscribed topics: owntracks/+/+ owntracks/+/+/event owntracks/+/+/info owntracks/+/+/waypoints owntracks/moe/phone/cmd
[2024-03-18 22:12:13.516] moe published to owntracks/moe/phone :
[2024-03-18 22:12:13.516] {"_type":"lwt","tst":1710796273}
[2024-03-18 22:12:13.516] moe disconnected

[2024-03-18 22:12:14.231] moe connected
[2024-03-18 22:12:14.242] moe subscribed topics: owntracks/+/+ owntracks/+/+/event owntracks/+/+/info owntracks/+/+/waypoints owntracks/moe/phone/cmd
[2024-03-18 22:13:14.255] moe published to owntracks/moe/phone :
[2024-03-18 22:13:14.255] {"_type":"lwt","tst":1710796334}
[2024-03-18 22:13:14.255] moe disconnected

It doesn't matter if the phone uses WiFi or mobile network. Since it happens almost exactly each minute and the immediate reconnect works, it doesn't look to me like some network or server issue.

My iPhone stays connected all the time if I leave the app open. Running in the background it connects on demand (pushing location) and closes the connection afterwards.

Btw, I use MQTT over websockets. The MQTT broker sits behind an nginx reverse proxy.
Connection between app and nginx is encrypted and secured via client certificate.
The same is true for the connection between nginx and the broker.

[ckrey]

Probably nginx terminates the connection after 60 seconds of inactivity (https://nginx.org/en/docs/http/websocket.html)
Either increase timeout values or investigate if nginx can do keepalive

[da-mkay]

Yep, it looks like increasing the timeouts in nginx solved it for me 😉

[jpmens]

@nicholasmparker can you confirm this solves the issue for you likewise?

[da-mkay]

After increasing the timeouts to 1 day in nginx I monitored the reconnects a few days. Sometimes the phone kept the connection for 2h, sometimes for 1h or 30 min or 22 min. The reconnect usually happened within 0 to 10 minutes. And when the connection is lost I see the error from the first post. Maybe these are "normal" connection losses due to ... Android stuff 😉

But today I switched the MQTT broker to Mosquitto, without changing the nginx config. Now I see the error log every exact 5 minutes while keep alive is set to 900s in the app. Reconnect happens immediately according to mosquitto logs. I do not see this behavior when opening the iOS app for let's say 10 minutes. It stays connected all the time.

Any ideas where this new magic 5 minutes come from? This was not the case with the last broker I used (aedes based).

[growse]

You've not said what device you're using, and that 5 mins could be a device thing. Do you see anything in the OT log about why it dropped the connection?

[da-mkay]

It is a Galaxy S21 FE. The OT log shows the error from the first posts here (connectionLost error).

A few days ago I switched back from Mosquitto to my other broker, and the connection losses every exact 5 minutes are gone. Now I am back to the behavior where the Android app can sometimes keep the connection for 2 hours, sometimes for 20 minutes and so on, which I guess is normal.

I don't know what the problem is between Android app and Mosquitto, because my other broker works well.
I also noticed some problems between iOS app and Mosquitto that I do not have with the other broker: When I opened the app I always saw an old outdated location of a friend. I always had to restart the app to see the current location.

I can solve both problems by replacing Mosquitto (v2.0.18 btw) which is fine for me.
But it would still be good to know what's the problem with Mosquitto. Are there any known issues or incompatibilities with Owntracks and Mosquitto?

[jpmens]

I'm having a hard time following. What is "my other broker", is that HiveMQ?

We're not aware of issues with Android and Mosquitto over MQTT, in fact most of us have that combination.

Can you get rid of the websocket connection and use plain MQTT (port 1883, or 8883 with TLS)?

[ckrey]

When switching between differen MQTT backends make sure to empty queues on the broker end by either

• resetting broker (mosquitto: delete mosquitto persistence db)
• connect once with CleanSession flag set (modify settings in OwnTracks App or use another MQTT client using the same clientId)

[da-mkay]

I'm having a hard time following. What is "my other broker", is that HiveMQ?

It's my own broker based on aedes (will be released soon).

We're not aware of issues with Android and Mosquitto over MQTT, in fact most of us have that combination.
Can you get rid of the websocket connection and use plain MQTT (port 1883, or 8883 with TLS)?

That's interesting. Unfortunately I don't have permanent access to the Android phone, so testing that scenario is a bit difficult. Maybe I can find an old Android phone here and reproduce the issue.

When switching between differen MQTT backends make sure to empty queues on the broker end by either

• resetting broker (mosquitto: delete mosquitto persistence db)
• connect once with CleanSession flag set (modify settings in OwnTracks App or use another MQTT client using the same clientId)

I did the latter by setting the clean flag in the app and restarting the app, then switched back to clean=false and restarted the app again.