Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Software lock up after temporary signal loss #64

Open
StevenLawyer opened this issue Sep 2, 2019 · 92 comments
Open

Software lock up after temporary signal loss #64

StevenLawyer opened this issue Sep 2, 2019 · 92 comments

Comments

@StevenLawyer
Copy link

Hello. Flying a Disco with a large capacity battery mod. Using the Huawei E3372 modem with a 128Gb microSD for video over Verizon in the US. Here's the problem I'm facing. On two separate long range flights so far, I've had the 4g mods "lock up". Here's what happens: Somewhere along the flight I hit a spot where the 4g signal is lost (due to distance from an antenna, aircraft orientation, whatever). Normally, when that happens, when the craft gets back in a good reception area (say in a minute or two), telemetry and FPV comes back and my glympse feed starts updating again. But twice now, I've had where it never reconnects. The first time, the Disco was over an hour away. For an hour I waited watching the skies until it came home. Something in the Disco had locked up to the point that, even when it got back to me and began loitering, I could not reconnect on the direct wifi in order to land the drone. I tried controller off and on, etc. Nothing worked. After the 5 minute loiter, the Disco auto-landed with me never regaining control. I powered the Disco off and back on and, as soon as I did so, everything worked including 4g connection and Glympse. Yesterday, I had a similar experience when I was over an hour and a half into the flight. I had had two momentary signal losses where they had regained. On the third one, I never regained telemetry, FPV or glympse. The drone came all the way home (through the exact same areas it had passed through outbound that have great coverage) with the connection never regaining, with the exception that I got one solitary glympse update about half way back. This time, I was able to take control via direct wifi when it got back. Again, after I powered the drone off and back on, everything worked fine. I have eliminated heat as the likely cause (good ventilation, cool weather, etc). I am running continual video recording onto the microSD in the Huawaii modem. On the first flight (the one where even wifi connection didn't work), the video file that was being written at the time of the loss of 4g control was not closed properly--clear that all connection to the modem had stopped. I don't think it was a cabling issue, since a power off and on immediately restored connection. On the second flight (where 4g and glympse quit but I could control via wifi), the video continued to record to the SD card. Any thoughts on what is occurring?

@s0ftice
Copy link
Member

s0ftice commented Sep 4, 2019

Hi Steven. I replied on Slack to you on Monday already. Guess you did not see it.
There have been occasional reports where a reconnection via 4G is not possible. Can you try to enable the debug logging, this might help to investigate what's going on. Note down the exact time when a disconnection happens the next time.
See https://uavpal.com/disco/faq#debug

Are you on the latest firmware versions of Disco/SC2 and the modem?

I never had this issue myself.

SoftIce

@StevenLawyer
Copy link
Author

S0ftice. Thanks for replying. Yes, I'm on the latest firmware on both. I have enabled debugging. Should be able to do a flight today.

@dru180
Copy link

dru180 commented Sep 16, 2019

Hi guys, has there been any further developments on this issue ? I had exactly the same issue, my disco came home and loitered till the battery was low and did an auto land. I changed the battery, reconnected after 3 power cycles, it all finally connected again. I than tried to commence another flightplan, and an error message on FFP indicated something along the lines of not being able to load the flight plan.
I'll try running a debug as well.
Cheers.
Dru.

@s0ftice
Copy link
Member

s0ftice commented Sep 16, 2019

No update no. Waiting for people to share debug logs.
Just to make sure you understood: The Skycontroller 2 will not fall-back to Wi-Fi automatically. It needs to be put into Wi-Fi mode manually (double-press Settings again). It can also be powered off and on again to reconnect via Wi-Fi.

The problem described here implies that even that did not work, correct?

@dru180
Copy link

dru180 commented Sep 16, 2019 via email

@StevenLawyer
Copy link
Author

So, I had it happen again (I think) on a flight that lasted 4 hours and 5 minutes. Unfortunately, my battery table was a little off and the drone ended up running out of power and crashing in the woods on the way home. I did manage to get the debug file off of it and will send it along with the time at which I lost control signal. I definitely was in good 4G reception. Unlike the other time that I lost control, I continued to receive my Glympse feed and, when I retrieved the video, it continued to record video (the first time, Glympse stopped, wifi wouldn't reconnect when home, and video recording to SD in Huwaei modem quit at the same moment, leaving a corrupted video file). On this last occasion, I can't say for sure whether or not the wifi connection ability locked out. I tried to drive to rendezvous with the disco on its auto-return and think I got close enough to connect via wifi, but never could get the connection. However, it's possible that I was not close enough. I can say that the live video feed locked up even though I got telemetry for a while and then all control locked up, but I still had good 4g connection (according to Glympse feed). Of course, it's possible that was all do to zerotier problems.

I'll send the times and debug file.

@Nicktk23
Copy link

Nicktk23 commented Sep 29, 2019

If the logs are showing that zerotier have issue, would be nice to be added a VPN (client) connection option (conf.file) to Disco to make connection in same LAN Network with SC2 . If im not mistakeing Huawei e3372 have built in VPN client ...probably no config on disco will be needed...i have to test that.

@s0ftice
Copy link
Member

s0ftice commented Sep 30, 2019

Hi Nick, if a direct connection is possible, zerotier will not relay the connection via their cloud infrastructure. It will just facilitate the initial establishing of connection but all traffic is directly routed between the two devices. It can also do NAT traversal which is required for most carriers due to their internal network design. A normal VPN client does not do that.

Feel free to use your own VPN client, but it will not be as easy as you described it.

Now question to all who reported this issue, do you guys see a [D]irect or [R]elayed connection in Glympse?

@StevenLawyer
Copy link
Author

StevenLawyer commented Sep 30, 2019 via email

@marph43
Copy link

marph43 commented Oct 5, 2019

On my last 4G-flight my Disco experienced exactely the same error, as originally described by StevenLawyer: When it lost the connection to the 4G network it was not able to reconnect at all, also when the network coverage was good enough again (during flight-to-home). Additionally it was not transmitting anymore to Glympse.
Even connecting directely to the SC2 by wifi, when it finally showed up at the beginning coordinates did not work anymore (Fortunately it made an automatic circular landing which went well). A reboot of the drone fixed the problem.
I will run a debug on my next flight in case it happens again.

@s0ftice
Copy link
Member

s0ftice commented Oct 6, 2019

Yes, please post the debug log if you can.
The important question that needs to be clarified is, what is the cause and what is the consequence.

While what you guys describe is a loss of 4G connection causing the Wifi interface to shutdown, might as well be: Loss of all network interfaces that makes 4G disconnect in the first place.

Could it be that the battery used does not provide enough power? Are you all flying on stock battery? The reason why I believe this is not a software issue: No reports of the same on the Bebop2 so far, which uses the same softmod code and modules.

@marph43
Copy link

marph43 commented Oct 6, 2019

I will.

Hmm, interesting point. Indeed, I am using a battery mod as well and it is possible that it can't handle power peaks as well as the stock battery (higher inner resistance). What I do not understand though, how can this only lead to the shutdown of the wirless devices and not of the whole Chuck?

@s0ftice
Copy link
Member

s0ftice commented Oct 6, 2019

no idea :) but if you guys can find something that all of your setups have in common, we might get a lead. I never had that issue myself.

@marph43
Copy link

marph43 commented Oct 9, 2019

Thinking further about this issue, led me to the idea that it could maybe be due to overheating of the Chuck. This could be the case, because I was flying against rather strong wind for more than 30 minutes. Is there a way to log the core temperature of the Chuck?

@StevenLawyer
Copy link
Author

StevenLawyer commented Oct 9, 2019 via email

@marph43
Copy link

marph43 commented Oct 27, 2019

OK. So yesterday it happened again. Somehow. Not sure if the problem was the same or if the network coverage was insufficient. The situation was the follwoing:

I flew close to a mountain and tried to cross it, but by that time the battery was already half empty. When I stared to fly back home, it suddenly started to loose connection. At first it sort of connected again several times, but bandwith was insufficient to regain control and the video was very very choppy. After some reconnections it disconnected permanently. Because I have an additional GPS tracker installed, I saw that the Disco fought against the wind to return home, but I underestimated the wind at this high altituede and was now unable to pilot the plane to a lower height, so it was not long until the battery was depleated and it stopped returning. Instead it was carried out again by the wind and went down somewhere in the woods. Luckily I was able to retrieve it, because of the additonal GPS tracker.

So now the question is, was it unable to reconnect because of the bad network coverage at this high altitude or because of something else? Like insufficient power? An interesting detail is, that the Chuck was running until the crash (logged everything also the route).

Please have a look at the attached debug log file. I lam looking forward to read your interpretation...

ulog_debug_20191026150251.log

@s0ftice
Copy link
Member

s0ftice commented Oct 27, 2019

Great, I will look into the log later. One question, when you found the Disco and CHUCK was still running. Did you try to reconnect your SC2 while in Wi-Fi mode? That would give an indication whether the reconnection was not possible at all, or just the 4G reconnection.

@StevenLawyer
Copy link
Author

StevenLawyer commented Oct 27, 2019 via email

@s0ftice
Copy link
Member

s0ftice commented Oct 27, 2019

Steven, yes I know, not just for you, some more reports as well. Just trying to figure out if marph’s issue is the same or something else.
By the way, when you answer via e-mail - due to your signature - your home address and phone number gets immortalized on the public Internet...

@StevenLawyer
Copy link
Author

StevenLawyer commented Oct 27, 2019 via email

@marph43
Copy link

marph43 commented Nov 2, 2019

@s0ftice : Sorry, I still owe you an answer. I can not tell you for sure if the Chuck was still running, because I retrieved it 10 hrs after it crashed. But as I mentioned, the additional GPS tracker was able to transmit the position during the whole flight, but it's only 2G and another carrier, than the 4G modem, so the coverage might be different. On the other hand it did not reconnect, during its descent either, although the coverage should have been sufficient. So my gut feeling tells me, that it was the same reconnection error all along.

Anyways. Did you discover something in the log?

@Nicktk23
Copy link

Nicktk23 commented Nov 2, 2019

i did some test`s (about 6 times) on the ground..and i notice that while in 4G mod, if connection on disco is lost for 10 to 20 minutes ...wifi and 4g mod are not accessible anymore, CHUCK is still runing but you cant do nothing to regain control. Il try to extract log file.

@s0ftice
Copy link
Member

s0ftice commented Nov 3, 2019

That's very useful. Thanks, I will try that. If I can reproduce it on mine, it will be much easier than log files.

@marph43 not yet.

@s0ftice
Copy link
Member

s0ftice commented Nov 3, 2019

@Nicktk23 I can't reproduce your test. I tried the following

  • Disco and SC2 end-to-end connected over 4G - FreeFlight Pro showing the video stream (on the ground).
  • I simply turned off SC2
  • From my PC, which is connected to the Disco's Wi-Fi, I'm pinging the Disco to see if the Wi-Fi interface is still up (192.168.42.1)

After 1 hour, the Disco is still reachable.
At the same time, I also had a telnet session on the Disco open (PC via Wi-Fi) to interactively see the ulogcat - maybe this session prevented the interfaces from going down?

@Nicktk23
Copy link

Nicktk23 commented Nov 3, 2019

after your last post i did the test once again , i give more details about the test run.
1st step : plug the power from battery and insert 4G modem Huawei 3372h into CHUCK useing OTG cable and turn Disco CHUCK On

2nd step : turn on SC2 and connect normaly on Disco`s WiFI.
3rd step : double press Settings Button on SC2...after few moments SC2 and Disco are connected (as expected in 4G mod)

4th step : double press Settings button on SC2 to switch back on WiFi mod
5th step : turn off SC2....after 14 minutes Huawei e3372h was blinking (showing is disconnect from 4G network) , on WiFi Network scan Disco Parrot wasn`t available anymore.

Maybe these steps help`s .. this glitch works on my parrot every time (sadly).

PS. i didnt keep alive a Telnet session.

@s0ftice
Copy link
Member

s0ftice commented Nov 4, 2019

Why the 4th step? That would not be a real-life scenario, as when you lose 4G when flying, Wi-Fi will hardly be in range to reconnect immediately.

Can you repeat your test with out 4. ?

I will repeat mine without a telnet session.

@s0ftice
Copy link
Member

s0ftice commented Nov 4, 2019

I have been following your steps and after 30 minutes, I can still see the SSID in a Wi-Fi scan and the modem shows solid LED.
I also ran your step 4, even though it does not make much sense to me as explained above.
I also did not even enable telnet with the double-press on CHUCK's power button in today's test.

Maybe someone else wants to try?

@tuttoto
Copy link

tuttoto commented Nov 21, 2019

after your last post i did the test once again , i give more details about the test run.
1st step : plug the power from battery and insert 4G modem Huawei 3372h into CHUCK useing OTG cable and turn Disco CHUCK On

2nd step : turn on SC2 and connect normaly on Disco`s WiFI.
3rd step : double press Settings Button on SC2...after few moments SC2 and Disco are connected (as expected in 4G mod)

4th step : double press Settings button on SC2 to switch back on WiFi mod
5th step : turn off SC2....after 14 minutes Huawei e3372h was blinking (showing is disconnect from 4G network) , on WiFi Network scan Disco Parrot wasn`t available anymore.

Maybe these steps help`s .. this glitch works on my parrot every time (sadly).

PS. i didnt keep alive a Telnet session.

It seems you have everything working, but manage to obtain disconnection from the Disco (after a while, every time) while other users are not... do you have Glympse installed and active? (Or anything else able to keep the connection active?...).
Could you redo your test with Glympse and push notifications tool?

@tuttoto
Copy link

tuttoto commented Nov 24, 2019

I have been following your steps and after 30 minutes, I can still see the SSID in a Wi-Fi scan and the modem shows solid LED.
I also ran your step 4, even though it does not make much sense to me as explained above.
I also did not even enable telnet with the double-press on CHUCK's power button in today's test.

Maybe someone else wants to try?

Ok... I made the try... no telnet enabled. But I have the glimpse and push notifications options installed.
I followed the steps (including the 4th), and after 90mn was still receiving glimpse updates, my disco modem light was still solid, and I reached 12% on the Disco battery. Then, I started the Skycontroller2 and the hotspot on my Samsung S6 edge+, achieving : solid blue light as usual, controller seen in Auriga/FreeFlight, BUT no telemetry and no video streaming...
I didn’t touch the disco but switched the Skycontroller2 to WiFi, which gave me back telemetry/video streams.
Then I switched the Skycontroller2 to 4g and obtained again solid blue light, but this time I had telemetry and video streaming as usual and everything was working well.
My thoughts: the step 4 made the disco looking for WiFi and not 4g, while being visible for zero tier. Until the Skycontroller2 discussed with it in WiFi and told to it to go looking for data on the 4g subsystem. I can confirm that the 4g modem remained connected during the whole test, maybe because it saw no reason to disconnect or maybe because glimpse kept it “alive”.

@BigBear90
Copy link

This happened to me too today... Luckily It was not very far and somehow I managed to get a WiFi connection and land without video feed (controls worked even if video didn't)

@parrotdisc
Copy link

yes if you have that issue try to Flash it

@Leolux226
Copy link

i was thinking the same but not so easy flashig right HILINK firmwares on this sticks do you can shre us wich firmware your sticks have that work fine? i also baught a new stick E8372h but teh disco dont come to the zerotier with that modem light all good. I will check to find a HILINK E3372h-153 and truy agan hopefully i got fixed not so easy more to find this sticks

@ClassAxion
Copy link

@Leolux226 I can flash any e3372 -153 (h or s hw version) with HiLink, even the "bricked" ones. Here is my guide for flashing (including firmware that I use): https://www78.zippyshare.com/v/HV6C0dyE/file.html

@Leolux226
Copy link

i got 403 massage on that link :/

@ClassAxion
Copy link

@Leolux226 Ok, so try from this link: https://casmall1.smallhost.pl/E3372-GUIDE.zip

@Leolux226
Copy link

Leolux226 commented Nov 2, 2022

i just follow this manual with that link: https://www.youtube.com/watch?v=oRii6o6JeZI&t=287sand worked on first time i just connected sk2 and disco over 4g start video and lets see tha to all here

@parrotdisc
Copy link

parrotdisc commented Nov 2, 2022

in order not to go into the trouble of flashing it I have bought another modem with the web interface wich works fine... so I am not going to waste any more time to flash the one that is not working I rather throw it away

@ClassAxion
Copy link

@parrotdisc if you did, you would prove that flashing the modem solves this problem. I don't have such a problem because I only fly on Hilink.

@Leolux226
Copy link

I was now able to flash it. The tut on the Video works really good if you follow step by step. I think is a good idear to add this on the Faq that only the HILINK version work right. FInaly its fixed thx to all that helped i think this will help many guys for now and the future.

this is offtopic but is anybody able get glympse able to work right BAT volt dont show more thats no problem but if i fluy mostly only 30% of the route is inside. i recognise mostly and of the trip but even this is not compleed

@ClassAxion
Copy link

@Leolux226 Yes, tutorials like this will work for most people but if you got stuck with any problem during flash then you won't find a solution. My tutorial always work because I flashed many modems even those that were bought as damaged for $2.

@Leolux226
Copy link

thx saved im happy this problem is solved

@parrotdisc
Copy link

I have managed to flash the firmware of the not working modem wich has now web interface.
I will test it and let you know.
The test consist in just switching on the drone and leave it on the table connected to the 4g (no need to connect the sc2).
If after a few hours the modem has still a stable blue light and everything works this means the modem is fixed.

@parrotdisc
Copy link

when I go into the webui menu to change the language it stays in Russian, have tried also reinstalling the factory type
Google won't also translate the page
When I create a new profile inserting the new APN it won't even save it

@ClassAxion
Copy link

@parrotdisc Did you erase the user data? (last step in my guide) That's important.
Language you can change using the select box in right top corner.

@parrotdisc
Copy link

yes you are right... I forgot the last step now it's working! I will do all the tests now thanks!

@parrotdisc
Copy link

I have tested the modem with the new firmware and now it does not disconnect anymore. The 4g Connection was stable for 2 hours so I can reconfirm that the issue was caused by the firmware of the modem...

@ClassAxion
Copy link

@parrotdisc Great! I personally prefer hilink firmware for e3372 modems, I have no problem with them.

@parrotdisc
Copy link

Hi I don't know about the Hi link software the modem that had the issue did not open the web interface but was using an application instead that you had to install first to connect it to the internet (mobile partner) now after reinstalling the firmware is uses the web interface and works fine like the other two modems that I had tested already

@Nicktk23
Copy link

Nicktk23 commented Nov 5, 2022

This is great news, il upgrade the modem to Hi-Link, i had the same problem before. Il begin my own test. I hope this is the Fix ;)

@ClassAxion
Copy link

If it works for you, in my opinion we should consider the hilink firmware as stable option and non-hilink firmwares as unstable.

@Leolux226
Copy link

i baught more discos used here in germany and i saw that one of them have issue with the skycontroller even on wifi it get disconected. So its really improtant that your system work on bench like 1-2h without issues an dthan you can compare to the 4g connection. One of the worst things is if you restart the Sc2 there is no way get connection back without before connected over wifi its really improtent to know you cna restart all the lte stick the app but never restart the sc2 if the DIsco is to faar for an wifi connection.

@parrotdisc
Copy link

Hi there is a way to reconnect to the drone while it's flying even after restarting the SC2, since you have more drones connect it to another drone on wifi first and then switch back to 4g and it will connect back to the flying drone over 4g! I have a secondary chuck that I take with me ad use it for that when needed

@Leolux226
Copy link

if there is a way to cennect back after restart over 4g would be supernice i dont know why this dont work. Probebly is doining a initialisation or something. Probebly is possibel safe that from last connect for the 4g connect or something like that

@skyer29
Copy link

skyer29 commented Nov 20, 2022

è successo anche a me!! Una volta disconnesso dal 4g non è riuscito a ricollegarsi ne al 4g ne al wi fi. Glympse non dava più dati e posizione. Ha fatto RTH e atterraggio circolare .

@skyer29
Copy link

skyer29 commented Nov 20, 2022

In merito si è trovata una soluzione?

@parrotdisc
Copy link

ciao come già discusso il problema è causato dal modem dipende dal firmware dello stesso in alto nella discussione vedi la soluzione (agg firmware)

@skyer29
Copy link

skyer29 commented Nov 21, 2022

ciao come già discusso il problema è causato dal modem dipende dal firmware dello stesso in alto nella discussione vedi la soluzione (agg firmware)

aggiornare il firmware, quale versione è quella corretta? o modificarlo tramite pc?

@parrotdisc
Copy link

non so quale versione sia meglio io ho 3 modem e l'unico che aveva tale problema non usava l'interfaccia web quando lo uso per collegare il PC all internet..ma bensì richiedeva l'installazione di un programma che era nella memoria del modem.
Perciò dalla mia esperienza basta che abbia l'interfaccia web per funzionare bene.
I test li faccio lasciando il drone acceso sul tavolo collegato al 4g (non serve mandarlo il volo) puoi anche spegnere il telecomando. Se resta collegato per più di un'ora (io faccio test da 2-3 ore) senza scollegarsi il modem funzione bene.
Quello che non andava bene si scollegava dopo 30 min ma a volte anche dopo un'ora.
Dopo l'aggiornamento firmware ho fatto un test di 3 ore.
Nella discussione sopra trovi le note come fare l'aggiornamento.
In ogni caso posso garantire che il problema è causato da modem e non dal chuck

@skyer29
Copy link

skyer29 commented Nov 21, 2022

ti ringrazio mi sei stato di grande aiuto. Anche il mio al collegamento apre un app e non interfaccia web… ora devo solo capire come fare per rimediare (non so se basta fare un aggiornamento). Qual è il modello di chiavetta con interfaccia web?

@parrotdisc
Copy link

in questa discussione in cima c'è scritto tutto anche come fare l'aggiornamento...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests