chdir support for cross-project subninja

[Qix-]

ninja-build/ninja#1578
ninja-build/ninja#977

This is a proposed (and accepted but eternally unmerged) feature of mainline's subninja that translates all of the paths of the subninja to a root directory.

The way ninja does relative path resolution right now with subninja is a bit aggravating and makes it impossible to support chaining build systems in a generator-agnostic manner without severe performance hits.

[evmar]

I will think about this, but I am reluctant to diverge from ninja behavior when reading ninja files.

[Qix-]

This is planned behavior on their end, they intend to merge this. It's just that Ninja is nearly a dead project at this point (the milestones haven't moved in years). They also actively refuse extra help, so I'm not sure how to proceed aside from forking anyway.

My use case is that my generator would benefit from having third party build systems generate themselves (assuming they can generate to ninja) and then incorporate the nested project into its own build graph.

Currently, this is impossible with ninja. Subninja works more like include does since it's all local to the root ninja file (details are fuzzy on this, it's been a while since I opened the original issue).

If someone actually fixed this, a lot of very, very interesting interoperability would be possible.

[evmar]

I thought of a possible way to do this without introducing new syntax: what if you could specify multiple -C flags on the command line, and it would mean to load all the referenced build.ninja files from those (with the chdir semantics)? Would that address your use case? The one thing I'm not sure about is where the n2 output files would go, perhaps into the first specified -C directory?

[Qix-]

Not really because that would mean the user would have to specify those on the command line whenever they built something, which means if they missed one (of potentially hundreds) then all of a sudden their dependency tree is broken in super obscure ways.

[evmar]

Can you explain the scenario where you have a hundred ninja files that are not generated by the same generator? It seems like a recipe for bad performance in any case.

[Qix-]

Multiple build system interop with lots of submodules. "Hundreds" might be an exaggeration but the point still stands. It's not ideal but writing a parser for CMake, Makefile, Bazel, Buck, Autoconf, Meson, etc. just to have decent build graph integration is unfeasible.

[nico]

Evan: Think of this as not competing with the platonic ideal of a perfect build, but with the current reality of cmake builds shelling out to cmake and ninja in a subprocess since there's no good way to compose ninja files (and that's with a single generator in a single build system). I can't think of a better way than this approach to solve this, which is why I'm supportive of the upstream ninja change.

[evmar]

Another design question: are pools shared between projects or global? At least the 'console' pool (in Ninja's behavior) must be shared, but otherwise pools either need to be distinct because they will have different settings in different projects, or shared because the point of them is to limit work in parallel.

[Qix-]

The way it's intended to be treated is as if all of the ninja rules/builds were in the same build.ninja file but 1) had their paths adjusted (or had an implicit chdir prepended to all commands, something of that sort) and 2) were namespaced so that the rule names between files did not conflict. Everything else is shared the same as if they were part of the same build graph (in theory they are part of the same build graph).

So in this case console would be shared amongst all of them.

[evmar]

What happens then if two projects define the same pool name?

[evmar]

I think the current semantics of ninja are such that variables are inherited by subninjas:

// build.ninja
foo = bar
subninja other.ninja

// other.ninja
build $foo: ...

In the case of subninjas with non-communicating build systems I think you wouldn't want this, because other.ninja here would have been written under the assumption that foo was undefined. (?)

[Qix-]

Oh right, forgot about variables. Variables should be namespaced too, absolutely.

The goal is that this operates as if you called ninja on them separately, but are still integrated into the same dependency graph. This is distinctly different than invoking ninja -C from within ninja (as you might with makefiles calling make -C, which has a lot of problems), which would build whatever default targets are there instead of being intelligent about what is actually depended upon. Instead, if a nested build.ninja defines a thousand default build targets, but the parent only depends on one, then only one target is build from the nested config file.