Document how to set environment variables in subprocesses

[evmar]

See discussion on
https://groups.google.com/d/topic/ninja-build/ZGZ2Ewxsxaw/discussion
and previous thread linked there.

[nirbheek]

Reading through that discussion, it seems that environment variables aren't supported in Ninja because CreateProcess doesn't support setting PATH to find the executable to run when using lpCommandLine instead of lpApplicationName? The thing is, execvpe and execle have the same restriction, so I don't understand what the issue is here.

It's really simple to set the environment inherited by a process on all platforms, so I'd like to understand what the blocker is here. Cheers!

[nirbheek]

FWIW, we would really make good use of this feature in Meson: mesonbuild/meson#266, mesonbuild/meson#384, etc.

[nirbheek]

Would the project be interested in a PR that implements this? It's a troublesome blocker downstream for https://github.com/mesonbuild/meson/ since we have to add wrapper scripts for any command that needs environment variables set, which is slowly turning out to be quite a few.

Even Makefiles and Visual Studio project files support this, so Ninja stands out in not supporting this. :)

We'd be happy to implement this though if there is interest in having this feature in Ninja.

[nico]

On POSIXy platforms, you can do

rule foo
  command = ENV1=env1 ENV2=env2 my_command

or

rule foo
  command = $env my_command
foo out: in
  env = ENV1=env1 ENV2=env2

Is that not enough for you? Most commands shouldn't need env vars, and generally supporting less stuff keeps ninja simple.

[nirbheek]

Yes, that part is documented, but Meson is a cross-platform build system (meant to have feature-parity across all supported platforms) and we currently have to use Python script wrappers whenever we need to set environment variables or have arguments with special characters (especially newlines).

At first, we also thought that setting env vars is a niche use-case and pushed back when users asked for it, but a surprisingly large number of people need it. For instance, gobject-introspection in GNOME requires you to set CC/CXX/etc while invoking its tools, otherwise it uses an auto-detection mechanism that ends up using the wrong compiler. "Custom targets" in Meson can run arbitrary commands, and people often show up asking how they can set environment variables while running some tool that they can't change the behaviour of.

The Python script workaround is ok, but invoking the python interpreter for each command really slows things down sometimes.

[nico]

Maybe gobject-introspection could accept those via command line args instead? (In my experience, builds that rely on env vars instead of flags tend to be much more vulnerable to devs screwing up their local env, etc.)

MSVC's cl.exe requires a few env vars to find includes and whatnot; for that ninja has ninja -t msvc which has a -e flag that can load an environment from a file. So meson could theoretically use that on Windows, and the posix stuff elsewhere.

But using python wrappers for the rare, "needs to be flexible" case and making the common commands work without reliance on the env seems like the best approach to me.

[sam-github]

I think I've run into this issue. I'm trying to build a FIPS executable on window, and need to follow these directions:

https://www.openssl.org/docs/fips/UserGuide-2.0.pdf, section 5.3.2:

For the Windows®
 environment a perl script fipslink.pl is provided which performs a
function similar to fipsld for Unix®
/Linux®
. Several environment variables need to be set:
FIPS_LINK is the linker name, normally “link”
FIPS_CC is the C compiler name, normally “cl”
FIPS_CC_ARGS is a string of C compiler arguments for compiling fips_premain.c
PREMAIN_DSO_EXE should be set to the path to fips_premain_dso.exe if a DLL is
being linked (can be omitted otherwise)
PREMAIN_SHA1_EXE is the full path to fips_standalone_sha1.exe
FIPS_TARGET is the path of the target executable or DLL file

Even though FIPS_CC is defined in my environment, it looks like ninja is not passing the env to the child process (perl fipslink.pl ...). I'll see if -e works for me.

[nirbheek]

Another use-case for this is setting LD_LIBRARY_PATH before running executables built in the project. This is often necessary with executables linked to shared libraries built in the project to ensure that already-existing versions in a custom prefix are not used (-Wl,-rpath doesn't work on distros that set DT_RUNPATH instead of DT_RPATH such as Debian and Ubuntu). Autotools (libtool) uses shell scripts for this. It would be a significant slowdown of the build if we had to spawn a Python interpreter or run a shell script for such cases.

As another datapoint, cmake came across this use-case so much that they have a C wrapper for it which sets the environment and then calls the process. They need it anyway for their make backend, but ninja-only build systems would greatly benefit from this feature.

[evmar]

I'm not really sure what this bug is discussing anymore, since it looks like it was originally about updating the docs. :)

For LD_LIBRARY_PATH, setting it via the command line already works as Nico mentioned above. It doesn't involve another process; we must use a shell to parse the command line anyway. (Subprocesses on Linux are oh-so-cheap anyway; you can't even measure /bin/sh -c "echo hello" because it is so fast.)

On Windows we have ninja -t msvc. You might try writing your meson feature using that and then once we have some experience with that we could figure out how to improve it.

[marc-h38]

Kconfiglib has a hard $srctree requirement for out of source builds. It uses a few other variables:
https://github.com/ulfalizer/Kconfiglib/blob/424d0d38e7/kconfiglib.py#L108

we currently have to use Python script wrappers whenever we need to set environment variables

From: https://github.com/ulfalizer/Kconfiglib/tree/424d0d38e7be15c5#overview

The entire library is contained in kconfiglib.py. The bundled scripts are implemented on top of it. Implementing your own scripts should be relatively easy, if needed.

[nazar-pc]

I think I might have hit this in #1997 where Meson calls Ninja, Ninja calls Meson back, but without inherited environment variables and everything falls apart because of that.

[eli-schwartz]

https://mesonbuild.com/Reference-manual.html#run_target

meson has various cases where it runs commands which are "wrapped due to env", and sometimes, like for run_target(), this is because the created build rule has automatic functionality where it communicates namespaced MESON_* environment variables which are only valid in such a build rule, but are guaranteed to be there, be overridden by meson, and be correct. If ninja could set these automatically and portably without relying on wrapper python scripts, this would have a beneficial effect on build speed...

In my experience, builds that rely on env vars instead of flags tend to be much more vulnerable to devs screwing up their local env, etc

Given any environment variable set in build.ninja would be generated, statically coded, and always there (no ifs ands or buts), it seems unlikely to have real life failure cases. It doesn't matter how you screw up your env, because your env is always ignored and overridden by build.ninja

[Hi-Angel]

So… Doesn't this feature belong to the goals of Ninja project? The documentation says, quoting:

• very fast (i.e., instant) incremental builds, even for very large projects
  […]
• when convenience and speed are in conflict, prefer speed.

From reading the discussion, supporting this feature will improve build speed in many cases. So, is there any reason not to support it?

[wzssyqa]

I am not sure is it for my use case: run unittest of llvm with qemu-user.

Background:
To run a dynlink app with qemu user, we should set env QEMU_LD_PREFIX, for example

QEMU_LD_PREFIX=/usr/mipsel-linux-gnu

While ninja check still complains that:

mipsel-binfmt-P: Could not open '/lib/ld.so.1': No such file or directory

It means that QEMU_LD_PREFIX is not passed to subprocess.