Handling environment variables for local and remote builds

[jhdub23]

One of the most frustrating things about most build systems is that builds can be different from user to user, due to critical environment variable settings. For example Environment Variables Affecting GCC. I see problems with environment handling within buck2, as well (#532 and #466).

The scons build system has an environment management system which is a joy to work with. Since environment variables can impact the build itself, the build environment is also used as a dependency. If there is a change in the execution environment, the build signature changes, which is critical for binary caching integrity.

For example assume we start with a "-O2" build configuration:

env = Environment(CC='gcc', CCFLAGS='-O2')
env.Program('foo.c')

Later, change the build file to an "-O3" build configuration:

env = Environment(CC='gcc', CCFLAGS='-O3')
env.Program('foo.c')

The affected binaries would be rebuilt with a different build signature (O3 version of foo.o and foo instead of O2). Change the build configuration back to "-O2", and scons would pull the binaries from the first build (O2 version) instead of rebuilding.

What is the best way to handle something like this in buck2?

[JakobDegen]

So buck2 mostly does this right. When defining actions, rules can specify some environment variables to set, and buck2 will use this information to decide when to rerun or not rerun an action. Rules can in turn expose this interface to users via an attribute, for example the env attribute in rust_library. This isn't available on all rules, but normally not too hard to add

That being said, in addition to the "tracked" environment that is actually expressed in the build graph, actions also inherit what you might consider a default environment. This part of the environment is not tracked by buck2 and will not result in rebuilds if changed.

When using remote execution, the default environment is fully determined by the backing RE implementation, so you'll have to consult their documentation for what that default environment looks like. #532 was an example of breakage caused by different RE implementations making different decisions here. The local environment on a users machine should never have any effect on the build in this case.

Local execution, on the other hand, is currently un-isolated in buck2. Specifically, the environment that local actions inherit will be the same one that was used to start the daemon. Just like how buck2 does not enforce that you declare all your inputs, it also does not enforce that you don't depend on untracked parts of the environment. The Rust rules try and protect against this by explicitly sanitizing the environment before invoking rustc - that behavior is what #466 is about.

Sanitizing the local environment would be an important part of an implementation of isolated local actions.

[jhdub23]

Isn't #532 and #466 indicative that there are fundamental problems with environment variable handling? Is it possible to make the environment a first-class citizen? Copy the SCons/python strategy and make the environment a dependency, independent of the toolchain. The environment is the execution context of all commands.

env = [ environment_variables ]
any_rule(..., env=env)

The analogy is Python's subprocess.run(..., env=env).

Wouldn't that be a general solution that addresses all the issues with environment management?