Browsing database painfully slow on first load

[rwest]

This may be a consequence of some well-intentioned "lazy" loading (I remember programming something along this lines ages ago, to greatly accelerate booting the web server, which seemed a priority at the time of frequent development and debugging) but now often when I go to browse parts of the database, eg. https://rmg.mit.edu/database/kinetics/libraries/ it takes AAAAAGES to load the page (as it, presumably, loads the database into memory for the first time). With multi-threaded web server, each subprocess has to do this the first time it's called. It can be super frustrating when trying to use the website.

Writing this, I have deja vu memories. Perhaps it's largely done, and we just need to fetch
https://rmg.mit.edu/database/load
a few times (enough to hit each thread) whenever the server / service is restarted, or the database updated.
(Or maybe fix that method so it actually does what it's supposed to do, because I'm not sure that it does right now)

Or on a cron job?

Anyway, any efforts to improve response times as a user would be appreciated!

[mliu49]

Yeah, this is a big issue. In theory, we do fetch /database/load when we start the server for every single process, but it seems like it might not be working properly. There is also code which will reload the database if it detects that files have changed, which could also be malfunctioning and doing unnecessary reloading.

[rwest]

Yes, the unnecessary reloading crossed my mind too. I think it checks the timestamps of files?. I recall once having a setup where the networked file server's clock had drifted and was out of sync with the computer checking, so time stamps were always wrong; make would always rebuild everything.

[xiaoruiDong]

I am sorry for the inconvenience. As mentioned by @mliu, currently, there is a mechanism to check if a file is modified and reload the database, but it does not work properly. Even if no modifications has been made, it is still doing unnecessary reloadings. And thanks @rwest for the advice, timestamp check can be potentially problematic. I will do some diagnosis on that, and let you know if I find something.