Automatic V8 DEPS detection

[mmarchini]

I'm trying to find a way for us to auto sync V8 dependencies without needing to manually list them and without having to manually parse the DEPS file. One idea I had is to add depot_tools as a dependency to git node v8 (which the user can specify a path or we can download if not specified) and then use fetch/gclient sync to ensure dependencies are sync'ed.

The main reason I'm looking into this is that I'm looking into the idea of "hybrid builds" where we build V8 with GN (or use GN to generate gyp/cmake files), and for this to work without Node.js collaborators having to install depot_tools, we need to make sure all dependencies are fetched, including build and build_tools (see all submodules on rusty_v8).

[targos]

Note that I intentionally didn't use depot_tools in order to only fetch the dependencies that are required to build V8 for node (there are many other dependencies, some of which are large, that are not necessary to us)

[mmarchini]

We can have a ignore_deps if that's the case. Do you have stats on how much the size would grow with those dependencies?

[targos]

There are also the gitignore updates that are related to the deps that we use.

[mmarchini]

As a first step, would you be willing to add add //build and //buildtools dependencies, drop all ignores we have on the script, and add deps/gn which is updates alongside V8? We can probably drop depot_tools from our V8 testing script if we do that, and it opens the door to build V8 with GN in a hybrid setup.

Edit: also third_party/icu, I'm still working on getting this setup running so other dependencies might come up.

[targos]

I'm open to anything that works 😄

[mmarchini]

Cool! I'll try to get this working with node-core-utils, hopefully when we upgrade to V8 8.5 we can also have those deps added and then we can simplify the whole test-v8 setup.

Edit: and maybe in the near future we can drop the gyp files for V8 and have a gyp/GN hybrid build. What a dream 😄

[targos]

have a gyp/GN hybrid build

Are you ready for the nightmare it will be to make it work on Windows? :D

[mmarchini]

I'm ready to push super hard for us to drop msvc and support only clang like V8 does 😅

[targos]

$ du -sh buildtools build tools third_party/icu
85M     buildtools
631M    build
32M     tools
629M    third_party/icu

[mmarchini]

That's with the full fetch, which includes binary files and the git folder, those folder are considerably smaller when cloning a single commit

[targos]

This is what I get with gclient sync. Is there another way to use depot_tools?

[mmarchini]

I don't think so, but maybe we can git clean -xdf the repositories. Otherwise we'll need to stick with the approach we have today.

[gengjiawen]

those folders are git repo. git clean -xdf likely not to work.

[gengjiawen]

I like the idea. But I have bad experience both with depot_tools and gclient even on linux (setup and sync issue). Also git submodule actually is pretty mess up. Maybe publish related toolchain in a npm module.

[mmarchini]

those folders are git repo. git clean -xdf likely not to work.

The folders yes, but all the binaries that depot_tools downloads on the side aren't and can be deleted with xdf (and that's where most of the size @targos mentioned is coming from).

But I have bad experience both with depot_tools and gclient even on linux (setup and sync issue).

We all have 😅 but hopefully automating it via ncu would make things easier

Also git submodule actually is pretty mess up

I didn't suggest submodules in this issue :)

Maybe publish related toolchain in a npm module.

That could work, but we would face pushback from folks that want Node.js to be builable offline at any point for any commit after you clone the repository. Also, publishing the toolchain to npm has many challenges as we would need to store all toolchains for any commit. Of course we can reuse a specific toolchain version for many commits, but as the time passed the number of versions we have to store grows. This might not scale well either in terms of storage or developer experience.

I'm not saying we shouldn't do it, it's something I'm even considering as part of the hole "bye bye gyp" initiative, but the hole is a lot deeper than it seems. And what I described is basically us rebuilding depot_tools our own way.