no BLE traffic after a few days working ok

[frnandu]

I get no responses on /esp32radout/status when issuing commands to set temps, after a few days working without problems.
Can't seem to Rescan neither it just keeps on Scanning... forever. Reboot also doesn't seem to work.
Any ideas what might cause a BLE freeze on esp32 ( ESP32 DEVKIT V1 doit)?

[softypit]

Which version of are you using? The latest beta?
Do you have a console trace?

[frnandu]

Using v1.49-beta.
How do I get a console trace?

[frnandu]

I'm using it with 7 pieces of eq3 valves.
Also nearby is another esp32 with openMQTTGateway firmware for reading 7 pieces of Xiaomi MiJia BLE temp sensors, which BTW also freezes BLE communication after a few days (it's another board - m5stack). Do you think the amount of traffic on BLE might be causing these freezes on both ESP32 boards??

[softypit]

I don't think the BLE traffic should cause problems. I think a trace from the esp32_mqtt_eq3 unit may help to understand what is happening.
To trace the device just monitor the com/tty port that appears when the device is connected to a pc via USB. Use any serial terminal (e.g. hyperterminal on windows, minicom/gtkterm on linux or putty) with 115200 baud.
This should help:
https://docs.espressif.com/projects/esp-idf/en/latest/get-started/establish-serial-connection.html

[frnandu]

Do you need the trace just after connected, or during the BLE freeze? Since it freezes only after a random number of days, it's just not very convenient to leave a computer connected to it for a few days (have only laptop). Any other suggestions?

[frnandu]

Maybe I can connect it to a raspberry and somehow log the serial to some file....

[softypit]

You probably don't need to trace from boot but it is tricky to start tracing after the ESP is booted.
It should be possible to use a raspberry pi to monitor the serial output from the ESP.
You should be able to run putty on raspbian or you could try socat to trace directly to a log file.

[yunnanpl]

Dear all. With 1.42 I had two issues, one was losing WiFi and the second was "BLE freeze" (loosing BLE communication) (I needed to restart some of the ESPs on weekly basis).
With 1.49 the WiFi issues seems to be solved, but now the BLE freeze became a more critical issue.
Now, every now and then the BLE stops responding. I thought that it was due to my eq3 queries (every 2 minutes), so I changed it, but without any significant improvement.
Then I thought that some buffers/ram runs full, so I rebooted the ESPs twice a day. I thought it helped somehow, but not completely, so I started to reboot them every 2 hours... still this did not help.

So, the BLE communication randomly stops (it may be even faster than 2 hours after booting), so the MQTT messages from EQ3 also stop (no command work). Unfortunately, after this happened, the rebooting does not work either.

I have 4 ESP32 devices at home. 3 of the same kind and one different (with external antenna). The issue happens on all. I have also different power supplies (as I thought it might be unstable power source). The BLE freeze happens on all 4 devices.
Now, I upgraded to 1.51 and hope it will magically go away :D

I will look into it when I find some more time. I wanted just to let you know that is not only the frnandu's issue.

[frnandu]

I'm thinking about trying esp8266 + hm10 BLE addon to see if it has the same issues. Since esp32 uses the same antena for wifi/BLE it might be more complicated to have reliable usage of wifi+ble without freezes. Esp8266 and hm10 should be indepented hardware/antena so maybe there's less chance of such kind of freezes.

[softypit]

Have you managed to reproduce this on 1.51?
If so do you have a list steps to reproduce the problem? To understand what is happening a trace will be required. I may be able to set up a test but I need to understand exactly what commands you are sending before the problem occurs.

[yunnanpl]

I am using 1.56 beta now, and it is working very nice since 3 days (which seems to quite a good uptime) (the up-time counter is very useful !).
I have made some improvements to the antennas (a lot of aluminum foil), and I try to avoid sending mqtt commands simultaneously (at the same time) to more than one thermostat
(while switching all 8 eq3, I add 5-10 seconds delay between each signal).
So, my feeling is that low BLE signal strength and too many simultaneous requests might have made the problem. Now all the thermostats have at least -85 dB signal.
(getting a trace can be quite challenging, but if the problem will repeatI will try to obtain it somehow).
UPDATE: 4 ESPs, 6 days without issues :)

[Pasha1402]

Running 1.56 for a few month. 4 gateways, 5 controlled devices. one eq3 literally few cm from esp32. Mentioned above issue happens to all, longest no issue run time around 1 month, but sooner or later gateway stops communicating with devices and no longer sending anything in /radout/. While web interface still works and log containing rows commands supposedly sent to EQ3, but no responses in json brackets.

Another challenge is ESP no longer can be restarted from web interface, only hardware reset works (which is tricky for some of mine to reach physically). Reboot ESP option changes page to Rebooting for a while and then returns to front page with same Uptime in bunch of days, that clearly says it didn't reboot. Feels like frozen BLE / error prevents software restart somehow.

@yunnanpl's idea isn't workaround as tendency to freeze have also those gateways, managing only one thermostat, so surely only one command at a time

[frnandu]

I gave up on ESP32 and just use now a rasperry pi zero and its perfect for the job.

[Pasha1402]

@frnandu: what software do you use?

[frnandu]

eq3cli ( https://github.com/rytilahti/python-eq3bt )

[frnandu]

Also is running mijia temperature sensors and sending it on mqtt, and also works great.
I was before also using (besides esp32_mqtt_eq3) another esp32 with OpenMQTTGateway for reading the sensors and was having also the same issues of freezing after a few days/hours.
Now everything is runing on one rpi zero and I have zero problems :)

[yunnanpl]

@frnandu: yes, but it does sound as an overkill slightly :) I do have a few linux machines at home, adding 4 more (I have 4 esp32 to control 9 thermostats) sounds like a lot of linux updating.

That is true, the freezing also occurs on my systems, even I did upgrade the antennas even more (I soldered ~9 cm copper/silver wire which improved the BT signal by almost 10 dB in some cases). The whole system and BT communication became much faster.

So, now, as I have slightly more time at home, I got the serial debugging and stuff under control, but I need to find a way to save the debug log somehow somewhere (raspberry Pi ? :)). But I also teaching myself C++ and reading the softypit code. I also read about similar issues of esp32 and I found some suspicious things already.
It seems that handling the timer leads sometimes to "timer overflow" and system being stuck. Equally, some other people report issues with restarting of esp32, connected with timer issues.
In our case, it seems that the ble_operation_in_progress value is stuck, which also may disallow the restart, which also disallows to remotely correct the situation. For an unknown reason the watchdog also does not act, which was also reported in some other projects.

So... at the moment I am trying to setup the "toolchain" to compile the project with my fixes, but as I am really used to python, compiling c++ project looks like really rocket science to me :) as a first fix, I will try to add "enforce reboot" button ignoring the queued BT commands.

PS. Other possibility which I am also testing is using micropython on esp32 for this project.

[frnandu]

Are your 4 thermostats so far away apart that you need a BT-wifi bridge for each one?
My ONE rpi zero in the center of my 2 floor home is enough to reach 7 eq3s + 10 mijias spread around the 2 floors.
And I do prefer to admin and write code to a rpi than to an esp32....

[yunnanpl]

So, the WIFI is available around the house, but with 2 hotspots. There are 9 thermostats, clearly at the edges of the house. This is controlled by 4 esp32 = 2 esp32 on ground floor and 2 on 1st floor.
I do have a BT dongle with "huge" antenna on the server/NAS located centrally in the center of the house, but the range is just enough to communicate with 3 MIJAs and some other devices, also located centrally.
I was clearly hoping that spread BT "points" will allow me for nicer person tracking, light switching etc. Furthermore, spread ESP32, should allow me to connect temperature and humidity (and other) sensors, substituting MIJAs.

I also like python and Linux (with bash, etc) much more, but I feel that with microcontrollers like esp32, esp8266, arduino I can hardly go around C++ (whereas I know python, php, java script and other stuff). Maybe micropython will become mature enough, to allow me to easily make some other projects (gas sensor, carbon monoxide sensor, etc.).

Tough :) I was hoping for eq3 module for esphome too. But there is also some significant hurdle to start with it.

[Pasha1402]

Normal ESP32 (build in antenna) has about 3 time shorter BT range than RPI Zero WH. In my scenario (concrete walls, 9 floor condo flat, high interference) it is only about 3m. Followed Frnandu hint and replaced 2 ESPs with RPI - two days stable run, way faster message delivery and, of course, custom Python service does everything I want and exactly how I want (added heart beat and push updates when something changed on EQ3, that not possible with esp32_mqtt_eq3). The only downside is ~25eur cost of RPI with SD

[krmichal]

(added heart beat and push updates when something changed on EQ3, that not possible with esp32_mqtt_eq3).

I know maybe this is not the best place, but could you write something more about it?
Maybe you could share some code you wrote for the service?
I just discovered "BLE freeze" (I use ESP32-WROOM-32U with ext. antenna), which ruined all my plans for controlling EQ3 heating in my home... :/

[Pasha1402]

Sure. Extremely quick and dirty, I literally wrote it in 15 mins - no configs of optimisation, all hardcoded. :) Feel free to use, assuming you can handle python https://github.com/Pasha1402/EQ3_PythonTest

[yunnanpl]

So, as putting 4 (or more) RPi around the house, updating more linux systems (and taking care for their safety) is not a solution. I decided to rewrite this in micropython. I do not know C/C++ too much, and I do not really want to start. It is at the moment in no way as advanced as this project, but hopefully it will be. Python makes it also easier to add your own parts, without compiling, etc.
It is not quick, but still dirty :) It took me some time to make WIFI, BLE and MQTT stable and robust to survive broken connections, bad messages, etc.
https://github.com/yunnanpl/esp32_python_eq3

[Pasha1402]

So, as putting 4 (or more) RPi around the house, updating more linux systems (and taking care for their safety) is not a solution. I decided to rewrite this in micropython. I do not know C/C++ too much, and I do not really want to start. It is at the moment in no way as advanced as this project, but hopefully it will be. Python makes it also easier to add your own parts, without compiling, etc.
It is not quick, but still dirty :) It took me some time to make WIFI, BLE and MQTT stable and robust to survive broken connections, bad messages, etc.
https://github.com/yunnanpl/esp32_python_eq3

Great work! Have you consider adding OTA configuration changes and some basic commands, like "reboot" via MQTT payloads? Was one of the reasons for me to consider RPI in addition to better range - direct maintenance might be limited is some cases, like if you have half meter of concrete on top of the device :)

[yunnanpl]

Yes, this OTA stuff is planned. Still, at the moment there is WebREPL which allows it, thus not fully automatically, but to remotely update/change config. I do not have half a meter concrete on top of it, but all my devices are behind/over the furniture :)
So with WebREPL enabled, there is no need for direct maintenance anymore.

There are "reset" and "scan" mqtt functions already built in. Hopefully reset will be not needed anymore :)

The range is another story, but it is something I accept with ESP32.

[mrmatt001]

This is great stuff by everyone and I'm incredibly grateful people have shared their work. I've a few of these dotted around the house sending MQTT messages back to my home assistant. I've three valves that have been updated to firmware v1.46 and don't work - I'm assuming (from what I've read) that the ESP32s can't read /send the data as they've not been paired. Is there any way of pairing them in the new versions?

While I can't contribute code I'm more than happy to dedicate time to help anyone out. Thanks again all.

[yunnanpl]

@mrmatt001 Yeah... I saw there is an new firmware, and I know the old one works without any authorization... but I did not felt urge yet to upgrade the firmware :D
I do know how pairing work in Linux, but I have no idea yet how will it work in micropython, and I have enough work with the project already.
If there would be an easy way to downgrade, I would do it... but I fear there is no easy way.
EDIT: As expected, the pairing was added, but the protocol probably did not changed. Some people report that it works after manual pairing. Still, pairing on micropython might be not the easiest thing to do.
rytilahti/python-eq3bt#41

[MeisterQ]

Do you guys really have that much problems with the ESP32 Code?

Mine is working very fine.

Just sometimes, less then 2 times in 2 month i need to restart the device.

Im not sure if a script on a raspberry pi will do it much better?

Maybe another option would be a BLE node with Node-Red?

I was just hopeing, that, this firmware here on this repository would get more updates

[Pasha1402]

Do you guys really have that much problems with the ESP32 Code?

Mine is working very fine.

Just sometimes, less then 2 times in 2 month i need to restart the device.

Im not sure if a script on a raspberry pi will do it much better?

Maybe another option would be a BLE node with Node-Red?

I was just hopeing, that, this firmware here on this repository would get more updates

Success of this particular solution is very dependent on many factors: distance between devices, wall, level of high freq. noise, usage patterns, whatever else... if you only use it to infrequent parameters change in suburban house you have more chances. Flat in large condo in city center - dramatically less.

For example, I update thermostats based on external temp sensor every 5 min, plus shut them on/off per data from external door/window sensors (internal detection is rubbish) - for me BLE freeze was matter of 3 days to two weeks max. RPI with Python script runs for me since last Oct, for half year already with no single issue, trust it will for years non-attended. I was looking into debug logs for a month and then completely forgot about these "gates" existence. Even with RPI better antenna these is large number of disconnect/non-delivery errors, which if not handled correctly drive solution into non-operating state. The only RPI downsides - cost and linux updates, as mentioned above.

Therefore I expect much improvement from @yunnanpl micropython implementation - better libraries, easier error handling, debuging, etc.

Node-Red you would normally use as higher level application layer, what most of us do. If you mean using single BLE dongle for all thermostats, it is easier to use HA component then, but only works in case you have 5x5m studio and range limitation of BLE not the case.

[krmichal]

I've three valves that have been updated to firmware v1.46 and don't work - I'm assuming (from what I've read) that the ESP32s can't read /send the data as they've not been paired. Is there any way of pairing them in the new versions?

I'm not sure which version I have on my EQ3 valves - I just installed the newest one using "calor BT" application (yet it may not be the same version). It required pairing with my android phone, but didn't require anything on ESP32 with esp32_mqtt_eq3 - they work without any additional steps.

[yunnanpl]

@MeisterQ: I have 4 ESP32 around the house, each had around 1-2 week uptime, that means, that every 2-4 days one stopped responding. This clearly led to being too hot or too cold in this room, as I am not checking it all the time. Even if I implemented many counter-measures (home assistant waiting for a few seconds between mqtts, resetting all esp32 automatically twice a week overnight etc.). It also happened that some thermostat did not got the message... this is also bad.

The reasonable time for me to work without interventions is 1 year (of course assuming near to 100% command fulfillment). Above 1 year is great, below, is slightly annoying.
So now, all 4 thermostats worked 5 days, without restart, without any lost message. This never happened before, still, 5 days is no proof.

Concerning fixing issues of one piece, with something else... is just bad idea. One should not do it :)
I just saw how much Home Assistant code, and bash scripts I had to fix the issues... it is enough.

@Pasha1402: I also put my complete trust in esp32, so i use only temp control, but windows/doors are checked by sensors. The same with external temperature sensors (as eq3 has no readable temp sensor). That is why the Mijia (I have 2) is implemented, and some directly connected temp sensor will be added too (like BME280).

[AleXSR700]

Hello everyone,
I am running into similar problems with an ESP32 running tasmota and 8 eQ-3 TRVs.
The connection keeps breaking down and I cannot communicate with the eQ-3 anymore.

Is there a way to downgrade to Firmware 1.10? That version seems to be running well.

[yunnanpl]

@AleXSR700 : that should be no problem, just check the older releases and use the one which works best for you:
https://github.com/softypit/esp32_mqtt_eq3/releases
(not sure if you can downgrade with OTA, or you need to re-flash it).

[AleXSR700]

@AleXSR700 : that should be no problem, just check the older releases and use the one which works best for you:
https://github.com/softypit/esp32_mqtt_eq3/releases
(not sure if you can downgrade with OTA, or you need to re-flash it).

Thank you for your feedback.
The linked releases are the actual stock firmware? Because
I am not trying to flash an ESP32 dev board but downgrade the TRV firmware.

[yunnanpl]

Now I get it (clearly the esp32_mqtt_eq3 had no 1.10 release). Sorry for misunderstanding.
I have no knowledge of downgrading the TRV firmware... and as I read in the past that it can be hard/impossible, so I have not upgraded mine.
I know that the new firmware introduces more safety ("real pairing") for Bluetooth connection, which is likely to break older implementations (which do not expect this).

There is a topic for that:
#67

[Pasha1402]

Not exactly the answer to the topic, but it worth to give up EQ3 now. Cheapest zigbee thermostat already more reliable, has better thermal control and rich two way data exchange. Technology has it progress...

[AleXSR700]

I am staying away from proprietary protocols for now. I see no point in moving away from wifi. And since I can forward from bluetooth to wifi (and need to for other sensors also), I won't be swapping working equipment.
Zigbee and z-wave might be out the door next anyway with the new protocol waiting for full implementation.

So no thank you. I'd rather sort out the eQ-3. :-)