fix(plugin-kit): limit concurrent file descriptors for adapters (DT-1707)

[lihbr]

Context

On projects with high numbers of slices, file descriptors limit can be reached when reading user's types.

Fixes: nuxt-modules/prismic#201, DT-1707

The Solution

This PR limits the amount of concurrent file descriptors allowed to adapters through their shared plugin-kit/fs helpers.

Important

I'm not really happy with this solution because it only addresses the issue by targeting the most file-system intensive part of Slice Machine. Limit could still be reached elsewhere. Also, determining a limit that works across all OS's is hard and there's no handy package available to dynamically fix it based on the current OS limitations.

Impact / Dependencies

/

How to QA

Try using Slice Machine with a large amount of slices. To help with that, here's an archive containing about 400 slices which should be reaching the limit on most systems: 231025_DT-1707.zip

Checklist before requesting a review

• I hereby declare my code ready for review.
• If it is a critical feature, I have added tests.
• The CI is successful.
• If there could backward compatibility issues, it has been discussed and planned.

[linear]

DT-1707 AAUser, I don't hit the file system limit for opened files (EMFILE: too many open files)

nuxt-modules/prismic#201

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
slice-machine	✅ Ready (Inspect)	Visit Preview	Oct 26, 2023 1:49pm

[angeloashmore]

There are some solutions that handle EMFILE, like graceful-fs and fs-extra (which uses graceful-fs). However, graceful-fs doesn't include fs/promises.

Overall, the current solution seems okay to me. We could iterate on this PR with one of the following options:

1. Create a drop-in replacement module for fs that uses p-limit. Allow users to override the limit with an environment variable (e.g. SM_FS_LIMIT=256).
2. Create a drop-in replacement module for fs that uses graceful-fs and node:utils's promisify (see this comment).
3. Use fs-extra.
4. Abstract the calls to fsLimit() to individual helpers, like lib/readFile.ts and lib/writeFile.ts.

Of these, Option 1 seems the cleanest to me if we are confident a static concurrency amount will work.

If not, Option 2 would work better, keeping in mind the extra dependency and the extra work needed if we move Slice Machine to the browser.

Option 3 is okay, except that it makes us more dependent on other packages.

Option 4 would result in messy code, in my opinion.

I'd go with Option 1:

// lib/fsLimit.ts

import * as fs from "node:fs/promises";
import pLimit from "p-limit";

/**
 * The parsed number value of the SM_FS_LIMIT environment variable.
 */
const SM_FS_LIMIT = Number.isNaN(Number(process.env.SM_FS_LIMIT))
	? undefined
	: Number(process.env.SM_FS_LIMIT);

/**
 * The maximum number of concurrent file descriptors allowed to adapters to
 * minimize issues like `EMFILE: too many open files`.
 *
 * - MacOS default limit: 2560 (recent), 256 (old)
 * - Windows limit (per process): 2048
 */
const CONCURRENT_FILE_DESCRIPTORS_LIMIT = SM_FS_LIMIT ?? 1024;

/**
 * Limit concurrent file descriptors for adapters.
 */
const fsLimit = pLimit(CONCURRENT_FILE_DESCRIPTORS_LIMIT);

/**
 * Wrap a function with `fsLimit()` to limit concurrent calls. All functions
 * called with `wrapWithFSLimit()` share the same queue.
 *
 * @param fn - The function to wrap.
 *
 * @returns The wrapped function.
 */
const wrapWithFSLimit = <
	// eslint-disable-next-line @typescript-eslint/no-explicit-any
	TFn extends (...args: any[]) => any,
>(
	fn: TFn,
): ((...args: Parameters<TFn>) => Promise<Awaited<ReturnType<TFn>>>) => {
	return (...args) => fsLimit(() => fn(...args));
};

export const access = wrapWithFSLimit(fs.access);
export const appendFile = wrapWithFSLimit(fs.appendFile);
export const chmod = wrapWithFSLimit(fs.chmod);
export const chown = wrapWithFSLimit(fs.chown);
export const copyFile = wrapWithFSLimit(fs.copyFile);
export const cp = wrapWithFSLimit(fs.cp);
export const lchmod = wrapWithFSLimit(fs.lchmod);
export const lchown = wrapWithFSLimit(fs.lchown);
export const link = wrapWithFSLimit(fs.link);
export const lstat = wrapWithFSLimit(fs.lstat);
export const lutimes = wrapWithFSLimit(fs.lutimes);
export const mkdir = wrapWithFSLimit(fs.mkdir);
export const mkdtemp = wrapWithFSLimit(fs.mkdtemp);
export const open = wrapWithFSLimit(fs.open);
export const opendir = wrapWithFSLimit(fs.opendir);
export const readFile = wrapWithFSLimit(fs.readFile);
export const readdir = wrapWithFSLimit(fs.readdir);
export const readlink = wrapWithFSLimit(fs.readlink);
export const realpath = wrapWithFSLimit(fs.realpath);
export const rename = wrapWithFSLimit(fs.rename);
export const rm = wrapWithFSLimit(fs.rm);
export const rmdir = wrapWithFSLimit(fs.rmdir);
export const stat = wrapWithFSLimit(fs.stat);
export const statfs = wrapWithFSLimit(fs.statfs);
export const symlink = wrapWithFSLimit(fs.symlink);
export const truncate = wrapWithFSLimit(fs.truncate);
export const unlink = wrapWithFSLimit(fs.unlink);
export const utimes = wrapWithFSLimit(fs.utimes);
export const watch = wrapWithFSLimit(fs.watch);
export const writeFile = wrapWithFSLimit(fs.writeFile);

Listing each export is preferred over some looped reducer to allow for native ESM handling.

[xrutayisire]

Hey, following Angelo comment, what's the status of this PR?

[lihbr]

Hey @xrutayisire, it's ready to be reviewed.

[bapmrl]

Great work!