RFC: #2627: Fix template write EMFILE when writing a large number of pages

[kisaragi-hiu]

This fixes #2627. Caveats below, though.

When paginating a data file with a large number of pages, for some reason despite using graceful-fs when given a large number of files it would still end up throwing EMFILE. The error is emitted through TemplateWriter.js (generateTemplates), in an async, unbuffered loop to _generateTemplate → generateMapEntry (Template.js) → _write → writeFile (from graceful-fs).

$ prlimit --nofile=50000 npx @11ty/eleventy --quiet
[11ty] Problem writing Eleventy templates: (more in DEBUG output)
[11ty] 1. Having trouble writing to "_site/item/199999/index.html" from "./a.njk" (via EleventyTemplateError)
[11ty] 2. EMFILE: too many open files, open '_site/item/49141/index.html' (via Error)
[11ty] 
[11ty] Original error stack trace: Error: EMFILE: too many open files, open '_site/item/49141/index.html'
[11ty] Wrote 0 files in 93.93 seconds (v2.0.0-canary.16)

This PR limits the number of concurrent writes on Eleventy's side to completely avoid hitting the FD limit.

Caveats

I've set the maximum number of concurrent writes to 1000, just below 1024 which some systems use as the default FD limit. Actually reading the current process FD limit would be much better, as this causes an unnecessary slow down for projects that go above 1000 concurrent writes but stays below the actual FD limit. I don't know if there is a standard way of reading the process FD limit that doesn't involve shelling out.

Also, as shown below, for some reason using 1000 as the maximum number of concurrent writes still allows it to build without hitting EMFILE even if the FD limit is lowered to 512. My hypothesis is that this is a low enough number that it doesn't overwhelm graceful-fs.

Results

Without this patch, a 80000-entry project generated with SignpostMarv/11ty-eleventy-issue-2226@2e8f2da fails to build with EMFILE, as documented by #2226 (comment).

$ git clone https://github.com/SignpostMarv/11ty-eleventy-issue-2226 -b eleventy-2.0
$ cd 11ty-eleventy-issue-2226
$ npm install
$ node generate.js 80000
$ # using prlimit to explicitly set the FD limit to 1024 so this command can be run without checking system configuration
$ prlimit --nofile=512 npx @11ty/eleventy
[11ty] Problem writing Eleventy templates: (more in DEBUG output)
[11ty] 1. Having trouble writing to "./output/2.0.0-canary.16/80000/92e08c830313b35e9cc22f85492058672e3429bd250cdc99e9c4292408feb95d/7cc39edb-e5ae-4bfc-bef5-d613277ba061.html" from "./11ty/pages/test.njk" (via EleventyTemplateError)
[11ty] 2. EMFILE: too many open files, open './output/2.0.0-canary.16/80000/92e08c830313b35e9cc22f85492058672e3429bd250cdc99e9c4292408feb95d/2acfb91a-d3a0-4429-a6a5-e67157b9fa8b.html' (via Error)
[11ty] 
[11ty] Original error stack trace: Error: EMFILE: too many open files, open './output/2.0.0-canary.16/80000/92e08c830313b35e9cc22f85492058672e3429bd250cdc99e9c4292408feb95d/2acfb91a-d3a0-4429-a6a5-e67157b9fa8b.html'
[11ty] Wrote 0 files in 310.85 seconds (v2.0.0-canary.16)

With this patch:

$ prlimit --nofile=512 npx @11ty/eleventy --quiet
[11ty] Wrote 80000 files in 28.70 seconds (0.4ms each, v2.0.0-canary.16)

[zachleat]

HMM, this looks great!

I’m curious: the concurrency check implemented here is template-specific, which means that it will only fix EMFILE issues stemming from a one-to-many pagination template (not from many disparate source templates). Do we want a global concurrency check instead?

[zachleat]

This module cannot intercept or handle EMFILE or ENFILE errors from sync methods. If you use sync methods which open file descriptors then you are responsible for dealing with any errors.

For myself, I would also note the above quoted from the graceful-fs docs: https://www.npmjs.com/package/graceful-fs Just as a sanity check, we are using a promisified writeFile function from graceful-fs:

eleventy/src/Template.js

Line 770 in 4e8f1a8

return writeFile(outputPath, finalContent).then(() => {

so I think we’re okay here.

I’m curious if we should be filing this upstream in graceful-fs instead? I see some similar issues but most of the relevant ones are closed as fixed.

https://github.com/isaacs/node-graceful-fs/issues?q=is%3Aissue+EMFILE+sort%3Aupdated-desc

[zachleat]

Just thinking out loud here I wonder if the comment I left here is valid or not: isaacs/node-graceful-fs#160 (comment) Maybe using util.promisify with graceful-fs isn’t working as expected!

[zachleat]

I think the other possibility here is that this error might be specific to addPassthroughCopy and that graceful-fs is working fine for template writing?

[AramZS]

Digging into this approach and testing it out, but wanted to say that practically the file limit should likely be set to whatever ulimit is on the host system? Or whatever the version of that is for Windows, no?

[AramZS]

I confirm that this works in both the build and serve contexts it can be run in. This is strange to me and perhaps implies that there is likely a deeper problem in how Eleventy is handling things? I'm not sure.

[AramZS]

Thinking that this might be an issue with the use of fs instead of graceful-fs elsewhere I've tried to flip more fs calls to graceful-fs. (Test coverage not completed, but you can see what I'm doing here - master...AramZS:eleventy:switch-more-to-graceful-fs )

I thought this would cover most of the possible fs calls, but it did not resolve the issue like the PR here did.

[AramZS]

I did some further experimentation with running this over a longer period of --serve and found that, interestingly, this worked initially but given enough changes done in a row Eleventy will still crash out with FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory. I'll fiddle with the memory setup to see if it will also have the EMFILE crash occur given enough memory during the --serve rebuild.

[AramZS]

Continuing to test and @zachleat I am starting to think the same as you are here, that perhaps util.promisify is doing something unexpected in this process.

Also, worth noting that it appears the OSX max files open is 256. When I checked the setting on my OSX machine, 256 was still the default (12.6.1)

[kaleb]

I solved this in my own build by patching graceful-fs timeout to be greater than 60000. It is not ideal, but it works for me currently.

[epelc]

@AramZS @zachleat if you take a look at my recent PR I've switched all sync and fs.promise api calls to use graceful-fs async api. Should be a good step forward before potentially adding this one.

Graceful-fs can't catch the fs.promise calls and also doesn't handle sync.

#2915

*edited had incorrect link

[zachleat]

@epelc I have merged your PR! I think we’ll roll with that approach, as graceful-fs manages simultaneous writes internally (see isaacs/node-graceful-fs#3)

[zachleat]

Putting this one on-hold for now as we investigate the efficacy of #2915 to solve this one (currently on main, shipping with 3.0.)

[timmywil]

FWIW, I still run into this with large builds on Windows using eleventy@ 4501f58.

It may still be worth considering this solution as well.

[epelc]

@timmywil did you add the graceful da patch at top of your eleventy file. That’s required in addition to my patch.

#2915