Add buildDependencies to package.json

[rumkin]

I think it's time to move build dependencies out of devDependencies to a separated group. It could be named "buildDependencies".

Motivation

Development dependencies contain tools like Eslint, Webpack development server, static server and many others which isn't required on a build stage. This affects the final installation size and increases the time updated code spends in CI/CD. Also it creates extra load on disk and network to download and store megabytes of redundant code.

Also it lets developers to create all-in-one packages which contain build and development tools simultaneously. What affects the size of this packages.

Pros

• Faster CI/CD.
• More compact installations.
• Costs reduction.

Cons

• Such solution requires separation of build and dev configuration files for tools like Gulp or Grunt to avoid dependencies import issues.
• Requires Yarn, PNPM and NPM implement this feature all together.
• Requires learning to use new option or automation to prevent dependency misplacement.

[zkochan]

I'd rather avoid adding a new dependency type. Is there maybe an alternative solution?

[dominykas]

Have you measured how much time would be saved by excluding the dependencies such as static server?

[ljharb]

This seems like it'd be better filed as an npm rfc - that said tho, I don't think I'm convinced this would be helpful. When would you not want the build to run immediately after tests pass?

[rumkin]

@zkochan Thanks for participation, could you elaborate your position to work out weak points?

@dominikas I've calculated numbers for Webpack Dev Server. It has ~30M downloads. If each download takes only 1 second (on my laptop it took 5.47 seconds to install via PNPM) it would be about 8,300 hours or 347 days (per month). Also the package's size (including dependencies) is about 29 MB what gives ~870 Petabytes per month (for 30M downloads). Even if only a half of this downloads are made by CI/CDs it's about 435 Petabytes of pollution data and 173.5 days of wasted time. This is only one package. Why are you focusing on static server, any particular cause?

[zkochan]

I changed my mind. I am fine with additional types of dev dependencies field if @arcanis and, or the npm team will be on board.

[arcanis]

I'm not sure I really see it as a necessity. This is purely an optimization, it won't enable any new use case, so the bar has to be pretty high for us to adopt it. I'm not convinced it's the case right now...

Adding this new syntax will have a cognitive cost on our users - it's already tricky sometimes to split between dev and prod deps, now they have to deal with build as well. The CLI will have --production, but also --build. Even in terms of maintenance, it's likely more requests would arise (typesDependencies? flowTypesDependencies? windowsDependencies?), and saying "no" to these kind of proposals takes resources 😅

That being said, since from my perspective this buildDependencies proposal only affects the package managers that would end up implementing it (in that this field wouldn't have any effect once published), I think it's fine if any of npm or pnpm want to implement it unilaterally. I just, personally, don't think it's going to be very impactful, and I don't see it being added to Yarn.

(One last point: Yarn's already pushing you to cache your dependencies, so the download you mention doesn't/shouldn't really exist on our side. The main interest would be to preserve some disk space, but I don't see that as being a good enough motivation against the drawbacks I mentioned)

[zkochan]

Even in terms of maintenance, it's likely more requests would arise (typesDependencies? flowTypesDependencies? windowsDependencies?), and saying "no" to these kind of proposals takes resources 😅

That is why I think if it would be added (I am not sure it should be added), I'd vote for allowing random <any>Dependencies fields. And they would all be just devDependencies grouped separately.

[dominykas]

@rumkin I was thinking about this from the user perspective. You're saying it takes 5s on your machine, but that is probably installing just the dev server?

I just took a look at a fairly hefty frontend repo (react, webpack, webpack dev server, babel, etc), and it takes 40s to install using npm without any caching enabled. I doubt that just skipping the dev server would bring that down to 35s?

But I also doubt that bringing it down from 40s down to 35s is worth it (because in the specific case compiling webpack takes another 3 minutes and the tests take another 5 minutes). This is ofc anecdata, but it doesn't feel like it would be worth it even if it were to shave those 5s off the build time.

Bringing the install time from 5s to 4s, when you need to do all the other work, is also not a huge saving to mandate adding a new option to package.json that all the maintainers would have to support indefinitely, incl. the documentation and learning curve, etc?

[rumkin]

@ljharb I am not telling about running build steps without running tests. I'm talking to skip installation of things like dev servers, fs watching tools and so, which will be never in use into CI/CD. And this is what Vercel, Netlify, Travis-CI and other installs without the need.

@dominykas Yep we can compare this numbers with other to make it look insignificant. But if developers spend this time waiting for CI to complete and doing nothing (what I actually usually do), then it costs some money too. Even if average developer salary is $4,000 what gives us $20 per hour, then according to the provided numbers (173.5 days or 4150 hours per month) it could cost $83,000 per month or $996,000 per year for one package. I know it's no one's money, but it still money and someone just burn them. Maybe my numbers are invalid, then it's the question to us how can we check it. We are engineers and we should measure and count not just call things tough or hard to implement. So if my numbers are true, then it will worth to implement it, even if it would take 1 month of work.

Also as I've referred, I'm installing it with PNPM. It used cache intensively and downloaded 0 packages out of 15. So even if you use cache, you'll need a time to validate it.

@arcanis This could be an optimization. But as we all now Node.js is struggling of the overweight and bundle size problem. And we need to solve it. While I can suggest to rewrite package registry logic and update JS syntax and specification to solve it, it's near to impossible to convince people to migrate in a reasonable time. But it's possible to change it step by step. From some point of view a package registry itself is just an optimization over sending packages via email. So for me it's just one of required changes.

You've referred cognitive cost. I could agree it's important. But this is universal problem and it could be applied to any new technology or any enhancement. If it's hard to implement new features, then we definitely do something wrong and this is just an another issue. There're a lot of new developers who do things wrong while learning and we couldn't do anything with it. But there should be tools for those who want to do things in a right way. And then we need to think how to educate other developers to use this tools too. For example we can add interactive knowledge tests as part as developer documentation, like we provide readme file now. Or we can check whether package requires dev dependencies in production mode using require hooks.

In my opinion it's a mistake to mix dependencies like it is now. For example if I need different versions of Eslint in a single project, one as a dependency and another as an util for linting, I couldn't install both of them. While I can solve it by separating the project into two package.json files, and moving dev dependencies into top-level, like monorepos usually do. And this is where we can see that we actually have a project and a package here and this is two entities mixed in one file. So all devDependencies done wrong and this is why users couldn't figure out where to put them from the first sight. I'am adding this to cons section. But I think it's possible to create some automation solution here to solve most of the issues related to dependencies misplacement.

[ljharb]

In that case, it sounds like what you really want is to differentiate “CI” from “local dev”, and “build” is very much not the right way to do that.

That would require packages that have TTY-specific pieces to split those out into two packages. It would then require consumers to manage twice as many dependencies (for anything that cared about being a TTY). How would npm resolve these issues? Would this new category be recursive? Presumably so.

The cost of managing all this in CPU cycles on local machines imo far outweighs the trivial cost of download bandwidth or disk space, both of which are largely infinite and free (whereas CPU is finite and expensive).

[rumkin]

Seems like the last paragraph is confusing. Such separation is out of scope of this proposal and it was added for example. What I propose is just to handle dependencies in such order:

• dependencies
• buildDependencies
• devDependencies

And each section contains dependency exclusively, like it is now. Thus libs like mocha, typescript, babel, or webpack should be stored in buildDependencies and libs like webpack-dev-server and nodemon would be in devDependencies. But it all would work almost like it does now.

[ljharb]

@rumkin what are "build dependencies" tho? webpack dev server is not a build dependency, it's a local dev dependency.

Mocha (and other test frameworks) and transpilers (like TS or babel) and bundlers (like webpack) are required in every single non-production environment.

[rumkin]

What do you mean under "local dev dependency" then?

Maybe that would help. Today we have two kinds of dependencies into devDependencies. This is build dependencies and, let's call it, utility dependencies.

• Build dependency is any package you use to receive a runtime code, generate documentation or prepare package to publishing.
• Utility dependency is a package you use locally to enhance development process (Prettier, Husky, etc.), but require them into dependencies.

Husky is a package which patches git hooks. You couldn't install it anywhere expect of devDependencies, but you wouldn't use it somewhere except your local environment. And thus it is a utility dependency. So you don't need to install it on CI/CD server.

I propose to move build dependencies into buildDependencies and let utility dependencies stay into devDependencies.

[ljharb]

I mean, a dev dep that's only useful in local development, as opposed to in CI.

[ljharb]

As for husky, while I wouldn't suggest using it at all, you'd definitely want it used in CI - since in CI, git commits and pushes do happen.

[rumkin]

@ljharb

As for husky, while I wouldn't suggest using it at all, you'd definitely want it used in CI - since in CI, git commits and pushes do happen.

This question is out of scope. In my stack CI shouldn't commit anything, so I don't need to check such commits. I've already provided examples of packages which isn't required by CI, so let's use them as an example.

I mean, a dev dep that's only useful in local development, as opposed to in CI.

So you're agree there are dependencies which should not get somewhere except of local environment. Fine, so why do we handle them in the same way and why not to put this deps where they belong?

[ljharb]

I can't answer for everyone, but for me, because the extra complexity of doing so is not a worthy tradeoff for the relatively insignificant cost of some extra bytes being downloaded on install. Perhaps the tradeoff would be worth it for you.

[rumkin]

@ljharb According to @zkochan messages it's not extra complex. IMO it's not a rocket science or fully automated autopilot, it's just a utility on a well known (and pretty predictable) field. And as I've shown it's not only a question of extra bytes, it's time which costs way much more than anything else, and this time overweights time spent on implementation. But maybe you're right.

Anyway I don't like the opinion-driven development, this is where we could never find a proper solution, but only to practice eloquence. It's not the way global technologies should be made. We need numbers to make scientific decision and trials/modelling to choose the best solution. And I think it's a good topic for study or research. Does Node.js has connections with universities to make such research? Maybe it can fund it or help to find funding for such research?