Builder - Build-time argument passing (e.g., HTTP_PROXY)

[icecrime]

Description

In the past, we refused pull requests which were basically allowing to inject environment variables or such through the command line as it makes it too easy to write a Dockerfile which would only build assuming you know what it expects. We believe this would hinder repeatability of builds.

Nevertheless, there is a real need for build-time argument passing, one example which we often get being HTTP_PROXY. We believe Docker should have a predefined set of general purpose build-time variables it allows to override like this one, as well as a Dockerfile instruction allowing to define image-specific ones.

How to

The best description of the solution is described here #9176 (comment). However, we would still like to prioritize client-side build (#14298) before moving on to this feature.

To give a few more details:

• Introduce a new ARG Dockerfile instruction that defines a build-time argument. The instruction take the name of the argument, a default value (which can be empty), and an optional description.
• Introduce an internal list of builtin, Engine-provided, build arguments. In a first version, the set of predefined arguments will be HTTP_PROXY, HTTPS_PROXY, FTP_PROXY, and NO_PROXY.
• Introduce a new --build-arg command-line flag to docker build that will receive argument values (e.g., docker build --build-arg HTTP_PROXY=http://...) in a similar way that docker run -e works today.
• Build-time arguments will not be persisted into the resulting image.
• Build-time arguments are injected in the environment of commands being RUN'ed as part of the build.
• Build-time arguments are available as substitution throughout the Dockerfile (for example, they can be used as part of an ENV instruction).

What's next

Ping @duglin for review.
Ping @mapuri: we'd really appreciate if you could close your original PR and submit a rebased version of it that implements the spec described here. Thanks again for your patience 👍

Tentatively adding this to the 1.9.0 milestone.

[marcellodesales]

Expectations

@icecrime I think you definitely gave a great summary about the importance of this feature... As (#14298) is prioritized, do you see (#9176) also making their way to Docker 1.8??? I've been waiting for this feature for such a long time and it is blocking the development of a Docker Registry + Builder solution, for instance.

CI Servers in mind

Our worker hosts are behind the HTTP_PROXY and NO_PROXY, no HTTP connection at all from outside world... So, ALL of our Docker builds must depend on Images that are behind one of our Docker Registries. That is, all of the images depend on a parent that we host. Engineers trying out Docker for the first time usually complain about the inability of getting a Docker file from the public Registry and building it locally.

As we own our builder machines (those that runs "docker builds" upon git merge of a GitHub Enterprise environment), we could easily be augmenting the default build steps with the Host's options, which would include the HTTP_PROXY related environment variables automatically. All because YUM, NPM, MAVEN etc would need those variables in order to properly load dependencies.

Thanks and I would love get this ASAP...

[icecrime]

Added more details, and tentatively adding to 1.9.0 milestone.

[mapuri]

@icecrime thanks for pinging me.

At a high level, as I understand, this spec adds to #9176 a new ARG primitive to validate the passed --build-arg flags against.

I can start looking at changes to rebase my patch to this new spec. I will close #9176 and update this thread as I make progress. Just FYI, I will need a few cycles to scope the work and prioritize this among my other things so it might be a bit slow. Looking forward to contribute.

[icecrime]

@mapuri No problem at all, thank you so much for your understanding here. We will fix this ;-)

[thaJeztah]

@icecrime I think this will replace #4962 as well?

[icecrime]

@thaJeztah Yes!

[jgreat]

I have another use case for the ability to inject environment variables at build time: credentials.

We use docker to containerize our private apps. A lot of these apps need access to private package repositories like npm and github at build time. At this time the only way reasonable way to provide these credentials in an automated build is to include a file in the code repository. This isn't an ideal solution.

Having the ability to pass in an api key or username/password at build time is a feature I've been waiting for a long time.

[icecrime]

Build time secrets are an entirely different topic.

[mapuri]

@icecrime @duglin

just FYI I created the PR #15182 to address the requirements for ARG and --build-arg

PTAL when you get a chance.

[igorescobar]

👍

[brancz]

👍 This is also very useful to only include development and test dependencies when actually required, allowing them to be excluded in official images, which are used in production. Doing this currently requires multiple Dockerfiles.

[tpires]

+1

In some cases you have some Dockerfiles to support similar environments in which it just differs on the datasource, db_host, etc.

[thaJeztah]

@tpires that more sounds like runtime configuration of a container, not something that is needed during build; you should be able to use environment variables for that. (e.g docker run -e db_host=db1.example.com my-image)

[tpires]

@thaJeztah On Wildfly 8.x I need to add datasource(s) and for that I need to point it to one host. For now I've defined it as ${env.DB_HOST} and on the Dockerfile I've set ENV DB_HOST . But that host needs to exist and running my db. If I could pass that argument on build I could omit db_host, db_password, etc.

[xenoterracide]

I would like this to be available to ADD as well. I'm currently in a low bandwidth environment and what I'm trying to do is.

ENV FILENAME apache-archiva-$VERSION-bin.tar.gz
ENV MIRROR_ROOT http://archive.apache.org/dist/archiva/$VERSION/binaries/

ADD $MIRROR_ROOT/$FILENAME /opt

and I have a download of the file on my local filesystem, but that same build wouldn't work in a CI environment (I also wouldn't want to commit the tarball), so I'd rather that just go fetch naturally.

[duglin]

@xenoterracide I think this is a slightly different issue since you're asking for env var substitution on the ADD command, which isn't really related to this ARG stuff. I would suggest you open up an issue for it but keep in mind that per https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax it might be postponed (or rejected for now).

[xenoterracide]

@duglin I suppose I can do that, but it seems like the same thing, I need to define a build argument, but I want to do it for an ADD

[mapuri]

@duglin I think we can do what @xenoterracide is looking for with ARG (as build-args will work with variable substitution as well with changes proposed in #15182), please correct if I got it wrong:

> VERSION="v1.0" docker build --build-arg FILENAME=apache-archiva-$VERSION-bin.tar.gz --build-arg MIRROR_ROOT=http://archive.apache.org/dist/archiva/$VERSION/binaries/ < EOF

ARG FILENAME
ARG MIRROR_ROOT

ADD $MIRROR_ROOT/$FILENAME /opt
EOF

[duglin]

Ah, sorry, yes I know that ARG would work ok but what I forgot was that ADD supported env var substitution already - I thought we'd need to add it to the list of cmds.

[xenoterracide]

There is an additional problem for what I want to do, #16085, local and remote adds don't behave in a consistent way when dealing with tarballs

[calavera]

Closing this now that #15182 has been merged!

Thank you @mapuri 🤘

[thaJeztah]

@otheus hadn't heard of that one; how standard is it? If it's widely used, feel free to open a separate feature request

[otheus]

@thaJeztah ALL_PROXY is supported in curl since... well, the changelog doesn't mention, but mailing lists indicate at least since 2008. https://curl.haxx.se/libcurl/c/CURLOPT_PROXY.html (mailing list discussion https://curl.haxx.se/mail/archive-2008-01/0044.html )

Do we need a separate feature request to amend a bugfix?

[thaJeztah]

@otheus there's more than cURL, but please open a new issue to suggest adding ALL_PROXY to the default list of supported args

[alvinchevolleaux]

Is there some sort of prompt option when building a Docker image? I don't really like passing passwords in cleartext via command line. A prompt option in the build process would also make building more interactive and self explanatory without the user needing to be aware of many command line arguments before building. Of course for automated builds you could just pass some no prompt flag.

[VonC]

@alvinchevolleaux Sure, use docker-compose (1.6 with docker-compose.yml version 2 file and docker 1.10), and environment variables: http://stackoverflow.com/a/36084324/6309

[alvinchevolleaux]

@VonC But with this method you still need to write your password in cleartext right? And then you'll need to be sure to clear your history, etc. I meant a method where you can use a password prompt instead of setting env variables. I'm thinking of where I want to set the root password for a database container for example.

[VonC]

@alvinchevolleaux I don't know about a prompt method. For the rest, you are back to the old "Secrets: write-up best practices, do's and don'ts, roadmap" debate. Last proposal: #13490 (comment), using shsh exec: https://github.com/realestate-com-au/shush#use-as-a-command-shim

[DanielDent]

Currently, when a build-arg is used that was not specified in the Dockerfile, Docker fails the build with "One or more build-args were not consumed, failing build."

I'm using build-args to inject local proxies/mirrors in a CI build environment. While HTTP_PROXY and friends are covered by being allowed by default, NPM_CONFIG_REGISTRY, HEX_MIRROR, and similar variables for other package managers are not.

It would be nice to be able to inject environment variables for all supported package managers without needing to worry about if a particular build is using that build argument.

I don't entirely understand why this check exists. No similar check is done to see if a runtime environment variable is 'allowed' by a Dockerfile.

If there is a good reason for maintaining this restriction, perhaps a --build-arg-optional could be provided which does not fail if the ARG wasn't specified in the Dockerfile.

[thaJeztah]

@DanielDent please open a new issue; commenting on a closed issue is usually not the best for new feature requests

[innocentme1]

@icecrime I saw your previous comment saying "Build time secrets are an entirely different topic.". I am not clear on this point. ARG variables are not persistent in final images as ENV variables, so whats wrong in using them for passing secrets. Even the ARG documentation says that this cannot be used for passing secrets. - I am not sure why it cannot be used? Can you or someone please clarify?

Is ARG designed for passing variables like proxy etc. which are "only" needed/used for building images and not needed later? If yes - can't we just expand this for secrets too? Are these ARG's by anyway stored in final images?

[hairyhenderson]

@innocentme1 - while build args aren't persisted in the runtime environment, they are persisted in the image's metadata. See the output from docker history or docker inspect on an image built with these.

Also, I issued #24522 earlier to clarify the docs on this point 😁

[innocentme1]

@hairyhenderson makes sense now. Thank you :D

[rupakg]

@mapuri I see your comments from 9/25/15 above to @adamkdean question.

I think the current limitation comes from the fact that variable substitution is not supported for the FROM instruction in Dockerfile (https://docs.docker.com/reference/builder/#environment-replacement). Otherwise following would have been possible with the --build-arg flag in docker build, which I assume what @nuttycom is looking for, please correct me if I misunderstood.

$ docker  build --build-arg Version=$(git ...) - <<EOF
ARG Version
FROM base_image.${Version:-"default_version"}
...
EOF

I have a very similar use case as above.

Is there a resolution or plan to include ARG replacement in the FROM instruction? Is it because FROM has to be the first instruction that ARG cannot be supported for FROM?

Apart from var replacement solutions outside of docker (like using sed etc.), is there a way that others are doing it? Any feedback is appreciated.

[nuttycom]

@rupakg I did ultimately find a simple workaround for this - ARG values are substituted into RUN evaluation, so I'm now doing basically this:

ARG APP_VERSION
RUN echo "$APP_VERSION" > /opt/myapp/app_version

And then have a Makefile that I use to invoke Docker:

VERSION=$(shell git describe)

bin/server.js bin/test.js:
    haxe build.hxml

build-server: bin/server.js bin/test.js
    docker build -t "me/myapp" --build-arg APP_VERSION=$(VERSION) . 

[rupakg]

@nuttycom Thanks for your comment. I need ARG replacement for the FROM instructions which is purely a 'build' req. Maybe I am missing what you did? Can you show me an example of ARG with FROM? I would really appreciate it.

[nuttycom]

@rupakg Ah, now that I recall correctly, what I've been doing recently is actually something different from what I'd initially requested here. Sorry for the confusion.

[nuttycom]

@rupakg However, a combination of our approaches could work - what if you just used a Makefile (as I'm doing) and then used sed to replace the version number in the Dockerfile being streamed in via your << clause?

[rupakg]

@nuttycom Yeah, sed works fine. ARG is much cleaner but not supported. Thanks again.