Cant build a blog with thousands of pages

[Faiiya]

Describe the bug
I have a blog with around 12500 pages, all in mdx.
I have tried everything i can but it just refuses to build.
most of the time i get no error, others i get javascript heap out of memory, or yarn exit code 137
so seems to be memory related, i have tried to do it on vercel or with 16gb ram but no luck.
i have tried with a clean clone of this repo and just adding the blogs and it still fails.

To Reproduce
just clone the repo
create a big number of copies of a mdx article
and build

Expected behavior
i expect it to build

Screenshots

System Info (if dev / build issue):

• Windows 11 WSL 2 with ubuntu
• Node version 18.18
• yarn v1.22.19

[timlrx]

I guess you have tested it for a small number of mdx files and it works? I have not tested it on so many mdx files.

[Faiiya]

yeah with small number of files it works, but someetimes it breaks with maybe 500.
the maximum i suceed building was with 4000 in vercel

[ttys3]

I have almost 400+ pages, sometimes it failed to build on Netlify

[timlrx]

Might be worth instrumenting it to see what is taking up the most memory / build time. I suspect it is the pre-rendering done on Next.js side which makes it hard to optimize. Maybe at 1000 pages+ it might be better to consider using server side rendering.

Anyone has a publicly available repo with that many pages?

[timlrx]

Re-reading this again and 12500 mdx pages really scares me - how do you even manage that 😆

At that scale, you should probably consider using a database backed CMS.

[Faiiya]

what cms do you recomend? , i dont need any editing or special tools, just simply host them and thats it.

after they are on the cms i should be able to do ISR wich should hopefully fix my problems right?

[ttys3]

I only have 100+ mdx:

❯ fd .mdx | wc -l 
135

and some tags.

deploy failed due to build failure.

I check local, I think it is because it uses more then 8GB memory