Performance & Scaling Concerns

[gamemann]

With the creation and use of our private web scraper, we've already added 15000+ new mods to the website! This was a big accomplishment and we're already seeing a big traffic spike due to search engines like Google, Yandex, and Bing picking up the new mods added (with the help of many SEO optimizations I've made). I don't know what to expect in regards to traffic numbers since this is the first time I've created a project heavily relying on SEO, but I'm preparing for the traffic numbers to increase a lot over time.

This has led to a few concerns in regards to the production website's performance and scaling which I'll go into below.

Processor Core Count & Clock Speeds

Currently, the website is hosted on a VPS. In the future, once the website starts generating revenue after adding advertisements and receiving more traffic, I plan on purchasing a dedicated server that includes a processor with a high core count and clock speeds along with NVMe SSDs. I'd prefer DDR5 RAM as well, but we'll see. I may also try to explore cloud solutions, but they are normally fairly expensive compared to an unmanaged fully dedicated server which is what I'm used to. Though, scaling may be a lot easier with a cloud solution.

Disk Space

At first, all mod banners were stored locally on the file system. However, recently I've integrated and setup Amazon's AWS S3 storage via a bucket which seemed to be much more cost efficient than upgrading the server the website is currently hosted on which is a VPS that hosts my other websites as well.

Thankfully, S3 storage is very cheap and I currently only owe $0.03 this month after storing a few GBs of data along with many GET/POST requests already made (for storing/receiving images).

Note - Mod banners that were uploaded to the file system before utilizing AWS S3 still retrieves from the file system when shown, but in the future, once I confirm AWS S3 is the primary file storage we want to use, I'll move them over to the AWS S3 bucket!

next/image Local File System Caching

The only issue we have now related to disk space is the website utilizes next/image for most images excluding images from mod descriptions via Markdown. At this time, next/image stores external images (including images from our S3 bucket) inside a local file system cache and this builds up over time until we restart the Best Mods' Docker container.

The cache time for each image is typically 60 minutes (depending on the header value), but from the research I've done, there is really no way to disable this feature without creating your own package and copying from next/image (which is likely what we're going to have to do anyways).

Query Performance

I've been recently working a lot on improving and optimizing our expensive SQL queries. I've started documenting this process here. The main query of concern retrieves new mods as you scroll down on the Browse page since it retrieve relations such as sources and installers along with the mod rating (the most expensive operation, by far).

I've cut the original query's execution time from over one second down to ~200ms! While I believe there are still minor optimizations we can make, the margin of improvement is a lot lower now. Therefore, I believe everything else mentioned in this issue should be a higher priority in the future.

Scaling

At this time, the Next.js application itself is single-threaded. This isn't a concern as of right now, but we're going to want to utilize Next.js clusters which acts as a load balancer in the future.

With that said, we're going to want to scale our PostgreSQL server as well.

All of this will depend on whether we go with an unmanaged server or a cloud solution since cloud solutions usually offer their own scaling services. The former will take a lot longer, but I also feel it'd give me a lot more experience which motivates me.

I will add more comments to this issue over time when I start investigating each piece! If anybody has any input/suggestions, feel free to comment here as well! Any help/guidance is appreciated, but not expected also since it is my responsibility to ensure the website performs well and I will be doing my own research😄