Support for environment variables for building containers

[smysnk]

It would be nice to be able to use --env to specify environment variables that can be used in the docker build process. Currently environment variables are only supported as part of the docker run command.

An example use case for this scenario:

I use scp to copy files into my container as to avoid creation of unnecessary layers using the ADD / RUN commands. Sometimes I will change the DOCKER_HOST depending on docker networking setup (docker in docker sometimes uses 10.x.x.x range).

$ docker build --env DOCKER_HOST=172.17.42.1 -t foo .

Dockerfile
FROM private/base
RUN scp -r vagrant@$DOCKER_HOST:/vagrant/docker/foo/* . && /bin/bash /fs/build.sh
CMD ["/bin/bash", "/bootstrap.sh"]

[cpuguy83]

This has been discussed at large and very likely won't be added.
Introducing env in the build command creates host dependent builds.

This particular use-case would be better served by being able to remove or squash layers that contain build artifacts which aren't required for the final image. Such as #4232

[tiborvass]

@smysnk with the 1.1.0 release (coming very soon), you'll be able to have a builder image and do something like docker run -e FOO=bar builder_image | docker build - provided the builder_image when run sends out an image's tar (docker save).

[cpuguy83]

Woot, cool

[josh-devops-center]

Docker supports the ONBUILD instruction which allows for a delayed instruction to run. This allows for a template Dockerfile to be inherited and extended.

@cpuguy83 Why can't a similar mechanism be done? You say the following, though where is the discussion and what's wrong with doing something similar to ONBUILD? It's not clear why the "ONLY" option is to use env in the build command, rather than supporting this in the Dockerfile itself.

"This has been discussed at large and very likely won't be added.
Introducing env in the build command creates host dependent builds."

[cpuguy83]

@josh-devops-center I'm not sure I follow how else this would be supported in the Dockerfile.

I should say, I'm not necessarily against having support for envs provided to the build command, and also IANTM, just relaying from many past conversations around this topic.

The idea is that right now you can take a Dockerfile and build it anywhere without any extra knowledge other than the Dockerfile+uploaded context.
What one might do is provide an environment file that you expect people to change as part of the context, and source that for each RUN command.

[brettviren]

One use case to support this feature: I'm trying to use Docker to provide "standard" platforms for building our software in such a way that they can be exercised portably. However, site-specific things have to be worked around. In particular, setting the various "http_proxy" variables are needed on some but not all build hosts. A secondary use is to offer the resulting images for download and these should not ultimately bake-in the build host's HTTP proxy variables.

With "docker build --env-file=site.env [...]" I would be able to deal with the first use case easily by maintaining a build-host-specific site.env file. I'd still need a way to null these out (as a final build command) for the final image to support the second use case.

The only good alternative to adding --env-file/--env to "docker build" that I can see is to maintain my Dockerfiles with some templating system so that they themselves are built from a set of site-neutral files + site-specific definitions.

Thanks for any considerations on this!

[josh-devops-center]

@cpuguy83

The idea is that right now you can take a Dockerfile and build it anywhere without any extra knowledge other than the Dockerfile+uploaded context.
What one might do is provide an environment file that you expect people to change as part of the context, and source that for each RUN command.

This works. The ability to specify an environment file that the build then sources so that the environment variables are effective for all subsequent works nicely. We could then have a generic Dockerfile that acts as the template, and then rely on the local context for environmental builds which are repeatable. This capability be great to have in Docker.

[cpuguy83]

@josh-devops-center I'm not talking about new behavior here, just having a file in your project that each RUN command will (manually) source.

[josh-devops-center]

@cpuguy83

 I'm not talking about new behavior here, just having a file in your project that each RUN command will (manually) source.

How would this work for a template style approach as both myself and @brettviren asked for? I'm not looking to build the image until the environmental variables are set. In fact, in certain environment specific builds the image will not build successfully until the environment variables are all properly set.

I'm looking for a way to delay setting the environment variables, similar to the effect that ONBUILD has. This would then allow for a templated Docker build for different environments.

[brettviren]

FYI, today I played around with this. I'm still learning Docker so maybe what I'm doing is "bad". It's here: https://github.com/LBNE/lbne-docker

In short: I "ADD" dot.profile -> .bash_profile file and have this set any proxy env variables depending on what network it is on based on what is found in /etc/resolv.conf which itself is set by the host admin. Then, any commands that may need to download through our proxy get "RUN" with "bash -l -c '...'" which implicitly sources the .bash_profile and assures ftp_proxy, etc is set. It feels pretty kludgey, but so far it is working. Something cleaner would be welcome!

Also, as you may also notice, instead of templates I used a simple config file + docker-map/docker-py to generate the Dockerfile.

[charandas]

@josh-devops-center Did you get anywhere with your ONBUILD environment variables idea? I also think that it could be a reasonable way on controlling the build.

[thaJeztah]

For those following this issue; there's a PR for supporting build-time environment variables here; #9176

[freeformz]

It looks like #14634 brings this up again.

[skodumuru]

i have a Dockerfile in which i am running wget command as RUN wget --user $USER --password $PASSWORD ...... I am using docker-compose up command to make dockers up. Before this command i am setting as export $USER=skodumuru and same for password also. But this is not working. Is there any way of doing this without using shell script file?

[thaJeztah]

@skodumuru you can use the build-time variables, that were implemented in #15182. HOWEVER, keep in mind that passing credentials like that is insecure, and the credentials will be baked into the resulting image (or its "history"), so you don't want to share that image with other people if you don't want them to know your username/password

[skodumuru]

seems that build environment variables are for docker build command but we are using docker-compose command. how can i pass ENV variables with this command.

[thaJeztah]

@skodumuru I don't think build-args are supported yet on compose, but here's an issue tracking that; docker/compose#2111

[Jason-Grant]

I find a way to build containers depend on environment variables (actually they are build arguments), but work like environment variables. The bad thing about this method is that you can not use it when you build your containers by docker-compose, like @thaJeztah said build-args are not supported yet on docker-compose.

Here is the method

FROM php:7.0-fpm
ARG APP_ENV=local
ENV APP_ENV ${APP_ENV}
RUN cd /usr/local/etc/php && ln -sf php.ini-${APP_ENV} php.ini

you can run docker build --build-arg APP_ENV=prod . to build a production container.

[lrepolho]

Is there anyway I can replace FROM instruction with a --build-arg. ?

Example:

$ docker build --build-args region=us-east-1 .
// Dockerfile
FROM aws.ecr.huge.url.${region}/repo:php-apache
WORKDIR /var/www
RUN echo "@@@"

[marcbachmann]

Is there anyway I can replace FROM instruction with a --build-arg. ?

I'm currently using sed for this:

cat Dockerfile | sed "s/{{region}}/us-east-1/" | docker build -t php-apache -

with a Dockerfile that looks like that:

from aws.ecr.huge.url.{{region}}/repo:php-apache
WORKDIR /var/www
RUN echo "@@@"

[Alexhha]

Just alternative solution - using envsubst command from moreutils package

$ DOCKER_HOST=localhost envsubst < Dockerfile
FROM private/base

RUN scp -r vagrant@localhost:/vagrant/docker/foo/* . \
    && /bin/bash /fs/build.sh

CMD ["/bin/bash", "/bootstrap.sh"]

So the final command is

$ DOCKER_HOST=localhost envsubst < Dockerfile | docker build -t php-apache -

You can pass any number of variables

$ PARAM1=VAL1 PARAM2=VAL2 ... envsubst < Dockerfile | docker build -t php-apache -

[thaJeztah]

Docker 17.05 adds support for using ARG in FROM, so;

ARG MYTAG=latest
FROM foobar:$MYTAG

See this PR; #31352

[cpuguy83]

Is there anyway I can replace FROM instruction with a --build-arg

Yes, coming up in 17.05

[hatdropper1977]

@thaJeztah I tried your syntax and get an error -- Compose wants the FROM to precede the ARG. For example, in my Dockerfile I have:

 ARG es_version
 FROM docker.elastic.co/elasticsearch/elasticsearch:$es_version

I get this error:

 Building elasticsearch
 Step 1/14 : ARG es_version
 ERROR: Service 'elasticsearch' failed to build: Please provide a source image with `from` prior to commit

[thaJeztah]

@hatdropper1977 what version of Docker are you running, and what version of docker compose? Does it work correctly if you build using docker build?