Unable to handle many files (1000+)

[Revono]

Describe the bug

I tried to create a playlist with around 1000 mp4 files, everything starts to lag or does not even work or will take hours. It just loads for ever (feels like)

Upload worked without any problems.

To Reproduce

Try to create a playlist with 1000 files

Expected behavior

Create playlists with 1000+ files smooth

Desktop/Server/Software (please complete the following information):

Ubuntu 22.04
16GB Ram 4 cors
amd64
Server was not fully used

Config Settings:

nothing special and not relevant

Logging:

There is no error in the logs, i checked

[jb-alvarado]

Could you be a bit more specific about what exactly goes slowly or not at all?

I have just tried to work with a list of 5000 objects in the frontend. Drag&Drop is really slow, everything else seemed almost normal.

But I could imagine that it generally works better with a virtual list.

I will add it to my ToDo.

[Revono]

@jb-alvarado Sorry for to less information.
When i want to go into folders it loads ages and if i go to other tabs while the folder is loading it feels like the api behinde is frozen. Same happens when creating a playlist and checkbox this folder with 1.1k mp4's or check the "All" aswell there is no UX for creating a playlist thate takes a while but this should be never ever 10+ minutes.

i recorded some stuff for you, i think you will get it instantly:
https://photos.app.goo.gl/VjKhh7MMm9fZqsje7

btw. really like the idea about this software, really cool

[jb-alvarado]

Thank you for the explanation and the screen recording!

First of all: the errors from browser console are not important. It looks like that the engine is not running, so there is no connection to it. This you can ignore.

The thing with the file browser: yes that is a problem. I never had a situation where I had so many videos in one folder. The problem is, that the backend open every file with ffprobe to get its duration and by so many file it takes very long. On my local machine with a NVME SSD, 2048 file and all around 15-20 seconds it take by me ~30 seconds to open.

At the moment I have no idea how to reduce this time. The best would be some kind of pagination, but at the moment I have no idea how to apply this to a folder content.

At the moment I can only recommend to use sub-folders with less files in one. It will not fix the problem on playlist generation, there it will still take time.

The other issue with changing pages I can fix maybe with this virtual list.

[JunioCalu]

At the moment I have no idea how to reduce this time. The best would be some kind of pagination, but at the moment I have no idea how to apply this to a folder content.

Perhaps implementing some sort of hard drive cache file, containing the most recent information generated by the file scanning done with ffprobe, might alleviate the problem.