hiveaps don't survive autoupdate to new releases (from v2019/v2021)

[Djfe]

In Aachen we lost 100% of our Hive APs during migration to newer releases.
5x HiveAP 121
2x HiveAP 330

the 330s were lost on update from v2019 to v2021.
They are both installed at the same spot but we only managed to find one so far.
A powercycle was enough to bring it back to live.
We lost it again a couple days later on the update from v2021 to v2023.1.

The 121s updated just fine from v2019 to v2021 but were lost a couple days later, when we rolled out v2023.1 for them.

We haven't really gotten hands on the devices, yet.
But we wanted to let people know about this issue :)
If we get hands on, then we'll retest our migration and watch logs and we will also test v2023.2

[Djfe]

Update:
330 got a major raise in compat version on OpenWrt release 22
openwrt/openwrt@cfe79f2

I have no clue why they went offline then unless they weren't able to reactivate wifi (for meshing) after trying to install the sysupgrade. (which probably gets tried every hour)

[kpanic23]

Apparently it's a bug in the autoupdater: The Image check fails after all network has stopped, but network isn't started again after the failure, so the device is left without any network connectivity.
#2742 (comment)

[rotanid]

the actual issue is described in #2742 as @kpanic23 already wrote. you have to act manually on site.
@neocturne do you think the autoupdate might really be an issue here like @kpanic23 says? at the least the devices should continue to run with the old firmware...

[Djfe]

ok, then this issue will remain open for the HiveAP 121.
The question that remains for me: was this ever mentioned in the known issues section? 😅

[rotanid]

hm, this issue is simply not true as they "survive" it other than stated.
and if the issue with the 121 is the same, we don't need two issues for the same thing.
rather a new one about the autoupdater - if it actually is a bug there.

regarding known issues: you can answer it yourself by looking at the past release notes ;-)
at least the issue is marked as a known issue on github since 2023-10-27

[kpanic23]

... at the least the devices should continue to run with the old firmware...

Technically they do. The problem is that the autoupdater stops networking before the update, aborts because of image version mismatch and just exits without re-enabling networking. This results in the node fully running but without any network connectivity. I've removed the image from our update server, so the nodes don't get stuck every hour again trying to upgrade.
If this gets fixed in autoupdater, the fix should probably get backported to 2021.1.x as well, just for safekeeping.

[blocktrron]

@kpanic23 This was fixed with freifunk-gluon/packages@0648b2d

[kpanic23]

Ah, great!
I've just cherry-picked it into the 2021.1.x branch and started a pull request.

[rotanid]

@kpanic23 This was fixed with freifunk-gluon/packages@0648b2d

oh wow, and reported by myself over 5 years ago...
somehow we missed to backport it, d'oh!

[rotanid]

@Djfe can we close this issue?