RPI freezes

[tomtadatom]

I want to use the visualizer for piano learning by outputting midi notes from my DAW. Everything works until I stop sending midi, then the RPI freezes. Surely this is not desirable ! I was first using RTPMIDI over ethernet, and combined with the freezing, frequent loss of connection was encountered. So then I ordered the sevilla device, and things improved immensely, however when I stop my DAW transport to say, switch midi files, it freezes. Some things I tried were stopping wifi on the RPI (well at least attempted to, not sure if it worked), and some filtering of midi data like velocity and sysop messages. I suppose I could go to the onboard midi player you created but that would require ssh and I found the whole linux thing rather unsettling (As far as linux, the project was a great learning experience and you did great on the tutorial btw).

My DAW seems to require both input and output ports be enabled, so I select both ports from the pi. Only using 1 of 4 ports from the sevilla device. Really want this to work - I made two for my 88 and 49 keys pianos. Have an idea to narrow down the behavior, like rather then stopping the DAW playback, just try changings notes. Also perhaps some other midi player like synthesia. But any troubleshooting of the linux software would be above me.

Would be thankful for any help !

[ozturkkl]

I have a similar issue, I'm using RTPMIDI and after playing a while the led freezes randomly and the last pressed keys stay lit up.

Did you find a solution for your problem?

[tomtadatom]

Sounds the same as you describe with the last notes staying lit. Are you using the pi zero? I was tempted to purchase a more powerful pi. Haven't yet tried it with synthesia. I now have a komplete kontrol s61 which has leds built in, although it lacks the brightness of the led visualizer. Frustrating!

[ozturkkl]

Yes I'm using the pi zero WH. I am monitoring everything with ssh and it seems like the issue is with the code. For now, I solved it by setting up a cron job that restarts the visualizer.py every 30 minutes.

In the future I will look into the code and set up a segment where if there is no input detected for longer than 1 min, the code will restart (or whatever it's initializing that fixes the issue with the cron technique)

Also there is no error output in the console. I looked at the memory too so it's not like a memory leak that would cause failure over time. I think it's related to RTPMIDI connection because the app doesn't freeze.

[tomtadatom]

Good info ! I don't have the prerequisite programming ability to pursue your ideas but like the idea of the reset. RTPMIDI should be robust, since isn't it often used for midi hosting on the pi ? Are you running windows 10 ? Forgot if I'd tested on windows 7. Just pondering if rtpmidi is the problem.

[ozturkkl]

Yeah, I'm running Windows 10, I also believe RTP MIDI is probably not the issue here.

Another benefit of having the initialization or connection reset after not receiving notes for a while is...(or whatever is resetting the connection, there is a part in the code that does all the connections in the beginning)

There are still edge cases where the connection needs to be reset like waking up the computer from sleep or hibernation or simply restarting the computer but not restarting the pi or the visualizer.py. Or if you boot the pi before the PC, which is a similar situation. In all of these cases restarting the visualizer.py fixes the issue which means the initialization sequence was built in a way that it rebuilds the RTP MIDI connection.

And also I think the freezing is related to having the same connection over a long period of time because I've only encountered it when I start playing the keyboard after leaving for a bit. That is why I've set it up to reset the app every 30 mins and have never encountered the freezing since then. But this way has its own problem, if you are playing stuff and the time hits :30 it freezes while it's restarting the app.

So the ultimate solution could be to reset the connection (not the whole app like I do), and do this only when there haven't been any keypresses in a while (like 5 mins) and the frequency could be like every 10 minutes or as short as 1 minute if you want the connection made faster (like when you leave the pi on all the time and you want the connection made within 1 minute of pc boot).

I couldn't find any other solution to having the connection made automatically because as far as I can tell, the PI might think that it's connected, but the PC could be reset or went to sleep and back. Ofc, if there is a better way of making sure the connection is active (and I'm sure there is a tool in RTP MIDI to test the connection or whatnot) that would be the best solution.

God bless you if you read all this :D

[onlaj]

@ozturkkl @tomtadatom Do you think reconnecting ports every 15 minutes (of inactivity) would solve the problem?
I made a branch with quick workaround, but I can't reproduce this issue in my setup, can you test it?

https://github.com/onlaj/Piano-LED-Visualizer/tree/freeze_fix

[ozturkkl]

I will gladly test it, thank you for the quick work :D

I will disable my 30 min corn job and will test these scenarios:

• PI stays open, PC restarts, goes to sleep and back, etc.
• Both stay open but after waiting a long period and coming back, are there any freezes like before?
• Any other problematic edge cases caused by connection reset.

I might play around with the number and make it super short like 10 seconds for testing faster. But 900 should be good for most scenarios.

[tomtadatom]

Some more info albeit not at the level of this conversation - as was already mentioned that I first was using the wifi method of connecting and rtpmidi on windows would constantly lose connection, and sometimes connection was able to be restored without resetting the pi. Switching to the seville device improved things a lot.

So suspicion is rtpmidi, but was just wondering if a work around could be done at the windows side? It's a long shot but perhaps windows is doing something strange. But I guess this has been accounted by ozturkkl by the most extreme scenario like the computer going to sleep. So any mac users seeing this problem?

[ozturkkl]

@tomtadatom
Yes, you are right! Sometimes RTPMIDI will act strange but I realized that as long as the service is running (and it was paired with pi at least once), when the pi attempts to reconnect, the connection will succeed.

I tested out the reconnect_ports method and have found some interesting things:

1. This method definitely works re-establishing the connection (pc restart, lost connection, etc.)
2. I had to add .close() to the previous port or it would lead to more serious freezing issues since the app would receive the keyboard output twice. Unfortunately, this does not get rid of the port completely...
3. The old ports are hanging around, look at them by aconnect -l because midoPort.get_output_names() won't show them, because of this the library runs out of memory. I could not find a way to close the unused ports completely without restarting the app.

The current state is like this:

• There is no freezing and the restart is just what we need.
• It can only be done around 32 times on my pi since I couldn't figure out how to remove the unused ports.
• Actually, it was half of that around 17 but when I did rtmidi.MidiOut.delete() it was doubled. I know, weird...

@onlaj
If you can find a way to literally delete the port, we can call reconnect_ports() as many times as we want! I even tried 1s interval and there is no delay or anything that is being caused by it, it is just what I need to always have the connection. Of course, right now it would crash after 32 seconds if the interval is set to 1 second.

Here's a link to the rtmidi hack I tried -> MidiOut.delete()
And here is the link for the changes I did if you want to test it out yourself -> https://github.com/ozturkkl/Piano-LED-Visualizer/commit/653d942263b6a9730c0e46f7c188a84c46c98f5a

I would be very curious about how you would solve this problem, ports not being deleted.

[onlaj]

Why not just use port.close() from MIDO library?
I commited to freeze_fix branch, can you test it?

[ozturkkl]

@onlaj
Yeah, I tried that but unfortunately, that does not get rid of the port completely, it just removes the connection. 2. point in my previous comment tries to explain why.

So, I believe MIDO is using rtmidi but the .close() implementation does not fully close the port or garbage collect the resources that were allocated. I don't know why but that's simply the case. Still, it's better than before for my use case... It works when I put my pc to sleep and back. To get rid of unused ports I restart the app every 6 hours now instead of every 30 mins.

Of course, if we want to push this for everyone to use we need to figure out how to delete the port (see my prev. comment) and stress test it like I do, setting the interval to 1s and running it indefinitely.

Again you can find the changes I made here -> https://github.com/ozturkkl/Piano-LED-Visualizer/commit/653d942263b6a9730c0e46f7c188a84c46c98f5a

[onlaj]

That's weird, I just tested it and simply adding port.close() prevented ports from being duplicated.
Without port.close(): https://streamable.com/lps0sv
And with: https://streamable.com/t5skx3
In both cases it was set to reconnect every second.

[ozturkkl]

I just tried it again, the interval was not set to 1s on freeze_fix so I just changed that nothing else and it was the exact same behavior for me.

Maybe it happens because I'm using RTPMIDI?

If I comment out .close() this happens which is way worse like I mentioned in my comment before:

[ozturkkl]

@onlaj
I don't know how you feel, but I think we were on to something here. Having auto-reconnect would be dope.
But if you think it's too much work or not worth, we can close the issue...

[onlaj]

I mean, we can try to fix that, but it's harder when I can't reproduce the same issue on my side. I left it running overnight with reconnect every 2 seconds and I don't have any duplicated ports. https://i.imgur.com/p4gZcoW.png
Are you testing my code or your own? I know they work on the same principles, but there are some additional conditions in yours. Maybe that's what made the change?

[ozturkkl]

Yes, I'm 200% sure I was on your branch, plus, as I said in my comment, your code is the first thing I did since it was the first obvious solution (using mido.close()). Also, you can see the difference above. The output that has close() enabled does not connect the 128:1 -> rtpmidi : KOz-Pc to every new input but only the last one.

But since the ports are closing on your side and you ran it all night without memory issues, that tells me it is possible to employ this solution for re-connecting. I'll try to look into it later and try to come up with a solution that actually closes the ports when using rtpmidi.

I'm also wondering why my output is different than yours. My PC is under 'rtpmidi raspberrypi' yours is under 'mio', maybe that's related? Are you using midi cable instead of rtpmidi? If so, would you mind installing rtpmidi and testing it out like that to see if the same thing happens?

And there's this hack I tried using rtmidi library that doubled the number of ports I could create before running out of memory so it did some amount of garbage collection. I am pretty lost but will try and give it another go, I love the app and would love to have it even more seamless with this feature. Thank you for taking your time and answering the thread.

[onlaj]

I'm using both sevilla's usb-usb and RTP midi. Last night I left it connected via rtp midi to w10 pc which run 24/7. Now I switched to mac os (also via rtp midi) which goes to sleep, get restarted etc, will see how it works with that. I will also try to physically disconnect the sevilla's device, just to make sure it doesn't change anything.

edit. after first restart I had to force "connect ports" to reconnect to rtpmidi, and I can see some duplicates, but under subports https://i.imgur.com/m7N7CRF.png

[ozturkkl]

Hey @onlaj

I have another thing to ask

• I realized that setting the interval more than 600 seconds won't work. The reason for that I think is the app goes into another loop, the screen saver loop. What should we do to get around that, I don't like messing with multiple modules in a well-structured code and rather would prefer you dealt with this since you know better.

I haven't looked at the duplicated ports issue yet but started using the feature. I restart the app every 5 hours to clean the ports. I will disable the screen saver on my branch for now.

Also, I still think there has to be a way to clean up the ports and re-create them without stopping the app, I will look into this later.

[onlaj]

What should we do to get around that

I disabled screensaver (set timer to 0) for my tests. Later when we figure out how to reconnect ports we can just add the function to screensaver's loop.

[ozturkkl]

I got updates for you guys!

I found the reason for my freezing issues :O They were related to my router deciding to change local IPs very frequently. _('. .)_/

The good news is, after setting both the pi and the PC as static routes, there are no more freezing issues.

The restart feature would be super nice for people wanting to never power off the pi and hope to reconnect after the restart but I could not find a way to close all the unused ports so it is not possible to do that at the moment.

Could you add to the readme if someone wants to use rtpmidi they should probably set static IPs for both pi and the PC @onlaj

[ozturkkl]

Scratch that, after longer testing, still got the freeze :'(

[tomtadatom]

ozturkkl - Still at it, huh? I wasn't aware DHCP was used for usb - so rtpmidi gets assigned ports dynamically? Makes sense though. But that didn't pan out either though, right?

So is it that only some configurations fail? Not everybody is getting the freeze problem, right? But certainly at least two people are. Although I have two separate pi's freezing, so that makes at least 3. Anyway, if a fix one day comes along, I'll be happy; but I'm just using the thing as a keyboard light now - not a total waste. I was gonna use it as a learning tool but it's better to just memorize patterns when learning piano than follow lights. Not helpful though if you're looking for the visualizer effect.

One thing interesting is how the light animations seem to continue even after the pi locks up. Haven't looked at the code and I can't even claim to understand the tech behind the lights, but just wondering what error the pi is encountering. But you probably already confirmed it is the ports being used up. Afraid I can't be of any use on this !

Is it possible to set up a debug mode? It's been almost 40 years but I remember setting trap points in the code. Of course it was hardware microprocessors I was debugging, hence needing the ability to single step through code.

[ozturkkl]

@tomtadatom
Yeah no luck, unfortunately. It usually freezes when I leave it for a bit and come back to play something, it works for the first 20 notes or so and then freezes.

I was using it as a visualizer so it sucks :/

Usually debugging output is just sent through the default output stream and I can't see any errors there. RPI does not freeze and I can go through the menus and when I reset the connection it starts working again so I'm sure it's something wrong with the RTPMIDI connection but I don't know what at this point.

The ports being used up thing was created when I tried to programmatically restart the connection every once in a while which solved the freezing but new connections would create a memory leak basically.

I thought my router was doing something to the connection, I still think it might be the case, like an automatic garbage collector for the network or something. If I get a chance I might try with a different router but idk when that will happen :-)

[tomtadatom]

Are you using the seville device? I tried both rtpmidi client and the seville which is class compliant. Both fail but seemed to be worse over the modem.

Thanks for clearing up the freezing issue for me - so either freezing or running out of ports. Well, hopefully you solve it. Just not important enough to me to attempt a solution, although I enjoyed reading your posts.

[ozturkkl]

@tomtadatom Hey man hope you're doing well.

I just wanted to update you on what I was up to this weekend. I was still having freezing issues and have been using the PI as a glorified static LED strip power source...

I think the issue is with the rtpMIDI software + some inefficient loops or connection code which I was unable to solve in my previous attempts. But definitely, not a problem related to this repo only.

I also think it could be my specific setup that is building something up and crashing eventually. Either way, I had it freeze on me while doing recordings a bunch of times and it was kinda the last straw before I started using the LED strip statically like mentioned above.

This weekend I had some time on my hands so I tried to implement my own little app using WebSockets. None of the rtpMIDI bonjour crap. Just finished the MVP prototype and it seems to be working splendidly. There's also a retry mechanism when the connection drops (which is inevitable when it comes to wireless). But it's super quick to pick it back up in the rare case of a disconnect...

Most importantly, I'm now able to step away for however long I want and immediately start playing and the LED visualizer works. Of course, my repo is nothing like this one and has every feature missing from it, even something as basic as picking the color with a simple UI is not there, yet...?

This was just a proof of concept project that I was curious about, turns out WebSockets are pretty good! Maybe it could be a motivation for Onlaj if he ever decides to rework the wireless logic. Until then, I will try improving my version with only the stuff that I need and just enjoy a reliable connection and working LED visualization (hopefully 🤞).

[onlaj]

turns out WebSockets are pretty good!

I'm already using WS to send pressed notes to Web Interface (for logging/debugging). Websockets are great, indeed, but the advantage of RTP is that it's already implemented on many systems. It's built-in on Macs and there are apps for Windows, Linux and Android.
Websockets would require additional client app for each system.

Good luck with your project, it looks promising.