Performance issues

[redge76]

Hi,

I get bizarre performance issues.
My lubelog instance is started in docker using almost a vanilla docker-compose.traefik.yml file.
I have about 20 other containers running without any issue. (AMD CPU with 32Gb ram. The data volumes are hosted on a SSD NAS)
My DB contains 3 cars and something like 80 fuel entries. (total)
The issue:
The urls that contains the data (for example /Vehicle/GetServiceRecordsByVehicleId?vehicleId=1 or /Home/Garage) can take up to 10s to load.
It's not always the case but about once every 10 requests, it take more than 3s.
I tried to bypass traefik but the result is the same.

Any idea how I could find out where the issue is ? I may try to migrate to a postgreSQL to rule out the integrated DB. (no time to do this now...)

Is there anyone else with the same issues ?

[hargata]

I can think of a few potential root causes:
Concurrency/DB deadlocks:
Do you have more than one user on your instance?
The flatfile DB(LiteDB) that comes with LL works like a charm with how efficient it is most of the time, the demo site relies on it and it gets hammered everyday. The only shortcoming I have encountered with the embedded DB is if there is a write operation at the same time as a read operation, which is a major concurrency issue and the biggest motivation for building out the postgresql backend.

Do you get any console errors when the slowdown happens? Usually it will throw an error log in the docker container about file locks and whatnot.

Recurring Reminders / Taxes
Do you have a ton of recurring reminders / taxes? If you have a good number of recurring reminders, are they all set to refresh / push back automatically when they become past due?
These are probably the most expensive operations done when loading the vehicle details:

• Calculating Reminder Urgencies and pushing back any past due reminders if they are recurring and the setting is enabled. (Used for displaying the yellow bell icon right by the reminders tab)
• Detecting any Recurring Taxes that are past due and automatically duplicate/refresh the entry.

But this doesn't really answer why the load is so slow on the Garage view since we don't load/calculate reminders in the Garage view until the user switches over to the Calendar tab, hence back to DB deadlocks.

Hardware Limitations
Pretty much just grasping at straws here, I really doubt hardware limitations can potentially cause such a huge slowdown on LL, the demo site runs on 1GB ram(yeap, seriously). LL is very lightweight and doesn't require high specs to run, BUT and this is a big one, the reason why LL can run efficiently on low-spec servers is because about 40% of the processing is done client-side, try this, press F12 or whatever key your browser uses to bring up the Developer console, navigate to the "Network" tab and check how much resources the web page is taking up.

This is on the demo site:

[hargata]

Adding to the Hardware Limitations potential root cause, what usually happens with modern web browsers, is that they will initially cache the resources fetched from the server and then periodically re-fetch the resource from the server instead of from the cache.

I know there was a user who uploaded a 20Mb picture of their vehicle before I imposed a limit on the demo site because the web browser had to download a 20Mb file just to display it in the vehicle tile and it was causing some massive slowdowns but only on the initial load, every subsequent refresh of the site it will pull from the cache and loads almost instantaneously, except for the odd request every so often when the browser decides to not pull from the cache.

[redge76]

Do you have more than one user on your instance?
Yes
Do you get any console errors when the slowdown happens?
no
Do you have a ton of recurring reminders / taxes?
2

Hardware Limitations
I doubt. i5-8400 CPU @ 2.80GHz (6 cores) + 32 Gb
All other containers have a good performance. Docker runs on bare metal.
I tried to move my lubelog to a local directory (was on a NAS mounted by FTP before) and there is no change.
As you can see on the screenshot below, there is4 requests at about 3.35s. (bizarre that they are exactly the same...)
I also tried to export/delete data dir/import the DB and there is no change.
There is something in my setup that lubelog does not like. Certainly something with IO/contention, but I don't know what...

[redge76]

OK I've taken the time to add a postgreSQL DB to my stack and I migrated the content of my LiteDB.
Here is the result. Not a single slow request.
(of course the DB is on the same system in the same directory as lubelogger. It is dedicated to lubelogger and it is started in the same docker-compose.yml)
:-/

btw, the migration went flawlessly. The only cumbersome step is to create the "app" schema. Why not just create if if not present ?
(usually most projets dont require this manual task)

[hargata]

Hey that's a major improvement! What's most likely going on is that your setup might have an overarching IO implementation that forces file locks on the flat database file so that it can only be read by one thread at a time and it forces all subsequent read/writes into a queue, but it seems like migrating to Postgres fixed the issue.

[redge76]

When my data files were on the NFS server, I would have understood. But I transfered all the files to the local SSD drive and the querie were still slow.
Postgres is great but I would prefer to stick with the default liteDB.
Do you plan to continue support of postgreSQL (It is certainly supérior but if I'm the only one to use is...)
Is there some kind of debug logging so I can try to trace where the lockup occurs ?

[hargata]

@redge76 there's actually quite a lot of users using the PostgreSQL backend, so it will be supported long term. I don't believe that the file lock errors get logged if there is an overarching IO preventing file locks as there will technically be no file locks to throw any exceptions.

I'm not sure if Docker have any IO implementations as such, but I do know that if you run the windows executable with LiteDB and switch tabs quickly you will get file lock exceptions, have never been able to replicate this on Docker so I am guessing there is something in Docker that handles file IO separately.