Slow initial start with huge collection

[eboye]

I've got a very big collection of music. When closing and starting the app it hangs for few minutes like this:

Is there a way it can load items from cache and sync in background?

This is the folder I'm using as collection in MusicPod

~45.000 files

[Feichtmeier]

hm yeah we could create an index or something
45k is something i didnt have in the beginning
I will think about this!

[Feichtmeier]

If it really a few minutes? Or seconds?

[Feichtmeier]

@eboye please test #265 or main after merge :) (or the snap from edge in some minutes, if you use snap ^^ )

[eboye]

Nope, I'm not using snap. I'm using the appimage.

First startup took around 7 minutes and it showed a message I've got 29k songs and if I want to save the cache. But after that the GDM broke and I needed to reboot.

I'm trying to start it again and it's rebuilding indexes again. Here's the screen recording I've made until the crash.

Screencast.from.2023-11-17.00-47-08.webm

I'll try to save the cache again once it's built. The crash may be due to recent update on of kernel modules and without reboot (I hope).

I'll report back the results if the cache saving passes.

I'll try to run the flutter project to have some logging if the crash happens again.

[eboye]

It didn't crash this time:

And closing takes roughly 5 seconds.

Next start is around minute or so with blank window while loading. I'll upload the screen record. But it's considerably faster.

[eboye]

OK, now closing is almost instant. Here's the startup screen recording:

Screencast.from.2023-11-17.01-05-58.webm

[eboye]

Here's the app profile startup. The first item in dart debugger says it took 11s. I think you can import it in your profiler or something. It may help.
dart_devtools_2023-11-17_01 11 53.406.json

[Feichtmeier]

Thank you for the detailed report!

I'll move the index building and index loading onto a second CPU threat (called isolate in dart) to not block the UI
Will ping you again when I'm done

[Feichtmeier]

@eboye okay :) check main again, I think this is now good for both extremes: very big library and cache wanted and people with small library or people wo dont use the local audio feature at all (and only use radio and podcast feature)

[eboye]

Well, it's kinda worse then the last version.

It acts like it's rebuilding every time even though it's set to use caching.

Screencast.from.2023-11-18.01-25-04.webm

[Feichtmeier]

Damn. Okay I think I need to rethink this totally Oop approach to audio files and their metadata. I think the problem might be the images in the mp3s

[Feichtmeier]

okay @eboye I think now it is better :) I made a mistake
to reset the settings delete .config/MusicPod/ or snap/MusiPod
slow machine today so slow video

Screencast.from.2023-11-18.11-42-53.webm

In my ~2k audio collection this cache slows down the start
But I think for 25k it should improve the start-up now (I hope)

if this finall works better for your huge library then I will set the threshold to ask for the cache maybe to 5k

[eboye]

Great! Not ideal, but I think this is much more usable :)

Screencast.from.2023-11-18.12-31-39.webm

I think we can close this issue. I have some experience with flutter (but mobile only), so I'll see if I can find some tweaks here and there to speed up the process. If I do, I'll make PRs :)

Thanks!

[Feichtmeier]

Thanks! This was after loading the cache?
If yes then hmmmmm I really wonder how to improve this. Eventually I should not store the in memory images in the cache at all. I need to rework the medata thing then. I have an idea but this will take some time
No let's keep this open as this is clearly still not acceptable
I guess other programs load faster even with 25k files?

[eboye]

Yes, with cache loaded.

Well, native ones are much much faster (C, Rust, Python...), electron is mixed bag.

Strawberry and Clementine are extremely fast for example. But the UI is garbage IMHO 😀

OK, but I really like the UI on this one, so I'll stick with it for now.

[Feichtmeier]

@eboye hey again. I've made another mistake and accidently created the cache twice... sooooo it should in theory be 2x fast for you now :) if you have time would appreciate if you re-check main / snap from edge

[eboye]

Will do. I'm using an appimage that I build myself 😁

I'll report back

[eboye]

Initial start is instant.

First start never finished indexing or what.
I've deleted .config/musicpod/
It started with loading icon (not sure why)
I've picked the folder. It indexed in a minute or two (much faster then previously)
Note: Scroll is short (for something like 50 items, when end reached it adds up)
Saved index
Closing takes ~10 seconds
Next start is instant
Loading of local audio takes almost the same time as first indexing (minute or two).

Is it parsing the index that long or is it reindexing?

When it finished, closing was instant.

[Feichtmeier]

Initial start is instant.

First start never finished indexing or what.
I've deleted .config/musicpod/
It started with loading icon (not sure why)
I've picked the folder. It indexed in a minute or two (much faster then previously)
Note: Scroll is short (for something like 50 items, when end reached it adds up)
Saved index
Closing takes ~10 seconds
Next start is instant
Loading of local audio takes almost the same time as first indexing (minute or two).

Is it parsing the index that long or is it reindexing?

When it finished, closing was instant.

Sounds good except that the loading of the local audio takes again same time. If I understand this correctly?

[eboye]

Yup, when the index is saved, loaded from cache, it takes some time, maybe not as indexing, but still it's slow to show something. I'll record the video if you need.

I expect it to be under 10-15 seconds.

[Feichtmeier]

Yup, when the index is saved, loaded from cache, it takes some time, maybe not as indexing, but still it's slow to show something. I'll record the video if you need.

I expect it to be under 10-15 seconds.

video would be nice
Really hard to test this because I dont have such a big collection. I could write a test for this maybe.

Make sure to delete the config folder of musicpod first

[eboye]

Hmm, I'm debugging now and it seems that ~/.config/musicpod is recreating when the app is started. But deleting it doesn't delete the index.

In ~/.cache/images are only images, there is no index files.

I look for MusicPod folder, but it doesn't seem to be in nor config nor cache. I'm not using snap, so there is no ~/.snap folder.

[Feichtmeier]

this seems to be different for app images then
in the snap it is one location
serch for settings.json or localAudioCache.json

[eboye]

Well, it is in ~/.config/musicpod

I'll try to investigate further and report back.

[Feichtmeier]

hey @eboye

I've cleaned up a lot of bad code. Especially today with localaudios could bring some improvements.
At least on a very weak machine I have here the local audio loading and searching is now much faster
Best regards 👋

[eboye]

Well, not sure if it's faster. This may be the issue:

the cache file is 2.3GB

In the file I see that you are caching images as base64 ... maybe you could exclude caching the images and load them lazy or something in the UI from the files itself?

[Feichtmeier]

Well, not sure if it's faster. This may be the issue:

the cache file is 2.3GB

In the file I see that you are caching images as base64 ... maybe you could exclude caching the images and load them lazy or something in the UI from the files itself?

Yeah.. it's sadly not possible to exclude the memory images while reading the metadata.
I think I need to work on this in the library I use for this

[eboye]

Well, not sure if it's faster. This may be the issue:

the cache file is 2.3GB
In the file I see that you are caching images as base64 ... maybe you could exclude caching the images and load them lazy or something in the UI from the files itself?

Yeah.. it's sadly not possible to exclude the memory images while reading the metadata. I think I need to work on this in the library I use for this

I've made a PR that can improve this, but then it lacks images later on

#467

[Feichtmeier]

hey @eboye
I want to try a different approach first
I switched away from metadata god package to the id3tag package
it promises to "Reading ID3 meta data without having to load the entire file into memory"
I have also added a test now which I would like to expand to your extreme test scenarios of 25k mp3 files.
Could you check out the main branch / snap edge and tell me if there is any improvements?
best would be if you would delete the settings files and your cache and not use the cache for the moment, I think the way I made the "cache" doesnt make much sense anyways