Add support for customizing the list of "default" build arguments supported by docker build

[TheFriendlyCoder]

Based on my current understanding, it appears that any custom build arguments provided to 'docker build' (ie: using the --build-arg parameter) must have an associated 'ARG' declaration in the dockerfile for the parameter to be used, except for several specific parameters which are somehow 'whitelisted' and thus do not require an associated declaration in the dockerfile. This is fine for one or two build arguments, or when managing just a small number of dockerfiles, but it quickly becomes unmanageable at scale.

Example Use Case:
Suppose a company has a mirror of a public package repository such as Python PyPI. The URL to the internal mirror can be set using an environment variable - PIP_INDEX_URL in this case - so that all Python package operations "just work" without further customizations (ie: pip install, etc.). Now, if one wants to create a Docker container which is able to install packages from this repository it would be beneficial to pass the relevant environment variable along to the container so it can be used at build time (ie: to be able to do a "RUN pip install" within a dockerfile, for example). To facilitate this behavior we can provide the relevant URL to docker at build time using "docker build --build-arg PIP_INDEX_URL=http://my_mirror", but in order for the dockerfile to make use of this environment variable we then need to add an associated ARG command to the dockerfile indicating that the argument is to be used. Further, since this is a site-wide customization, every dockerfile created must have this ARG command within it.

Now, suppose a company has 5, 10, 15 or more parameters used by automation and development alike, all customized on a site-by-site basis. As these permutations increase, trying to keep track of which dockerfiles use which variables where and when becomes tedious at best, and fragile at worse.

As a small step towards simplifying these more complex build environments, it would be helpful if there were some way to allow arguments to be pass along to a docker build without having to make modifications to the dockerfile being built.

Drawing some information from the extensive comment thread found here, maybe we could define a command line option, config file, or some such on the host used by the docker dameon to list one or more build arguments to inherit from the --build-arg parameter regardless of whether the dockerfile contains an ARG declaration for it or not.

Alternatively, maybe there could be a way to add a single boolean parameter to the docker build command to allow all build-arg params provided on the command line to be used regardless of the existence of ARG commands in the dockerfile being built - basically a flag that says "whitelist every build argument I'm giving you". This would allow us to override the default behavior of docker build on a case-by-case basis and would avoid having to make changes to the global environment. In our particular case we have a bootstrapper script that developers and automation uses to run docker operations providing an assortment of shortcuts and checks already. With a change like this we could simply update our bootstrapper script so the appropriate parameter gets passed along to the docker command and our users could reap the benefits without further intervention on their part.

[thaJeztah]

/cc @icecrime @duglin

[figroc]

I come across the proxy issue of docker build recently. I think #9176 is the right way to go for. Somehow it got closed in favor of #14634 which behaves in a case-by-case way. Then we need to find a way to customize the so-called "white-list".

[thaJeztah]

Build-args are passed to RUN commands as environment variables, as such, they also affect cache-matching (cd $HOME && echo foo > bar will produce a different layer, depending on the value of $HOME). Because of that, ARG needs to be used in every stage that needs to access a build-arg (and for that build-arg to be taken into account).

An exception was made for XX_PROXY build-args; while theoretically changing to a different proxy during build could give different results, the assumption is made for these that they are only needed to establish a network connection in the environment in which the daemon runs, but should not affect the result of the build.

In general, we try to make the Dockerfile describe the results of the build; requiring ARG's to be specified in the Dockerfile is part of that (and as mentioned earlier, is needed to allow cache-matching to work).

As to the original use-case; I wonder if that case can be resolved using the --add-host option, which would allow for custom hosts to be added to /etc/hosts in the build container (which would allow for DNS to match a different IP-address; which could be a mirror in this case).

[figroc]

I understand the rationale behinds those choices. But as long as there are resources from the network, you cannot guarantee that what you get is 100% consistent with what you described. These resources change over time, as well as the management decisions of the web servers.

We should rely on the consensus described, not the literal content. Changing '/etc/hosts' is the same as changing XXX_PROXY, as well as PIP_INDEX_URL, GOPROXY, npm_config_registry and etc.. We can enforce the consensus by not validating build caches against those environment variables.