New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Persisting ENV and ARG settings to all later stages in multi-stage builds #37345
Comments
Correct, Dockerfile instructions, including You can, however, set a global ARG version_default=v1
FROM alpine:latest as base1
ARG version_default
ENV version=$version_default
RUN echo ${version}
RUN echo ${version_default}
FROM alpine:latest as base2
ARG version_default
RUN echo ${version_default} When building without a docker build --no-cache -<<'EOF'
ARG version_default=v1
FROM alpine:latest as base1
ARG version_default
ENV version=$version_default
RUN echo ${version}
RUN echo ${version_default}
FROM alpine:latest as base2
ARG version_default
RUN echo ${version_default}
EOF This produces:
And with a docker build --no-cache --build-arg version_default=v2 -<<'EOF'
ARG version_default=v1
FROM alpine:latest as base1
ARG version_default
ENV version=$version_default
RUN echo ${version}
RUN echo ${version_default}
FROM alpine:latest as base2
ARG version_default
RUN echo ${version_default}
EOF
|
another way is to use base container for multiple stages: FROM alpine:latest as base
ARG version_default
ENV version=$version_default
FROM base
RUN echo ${version}
FROM base
RUN echo ${version}
|
Let me close this issue, because this is by design, but hope that the examples above help you further; also feel free to continue the conversation |
ENV vars and ARG are scoped per build-stage. A Workaround is a global ARG. see moby/moby#37345 (comment)
If this is by design, then the design is wrong. Without a concise way to share variables between stages it is impossible to DRY. There are unavoidable, uneliminable situations where many stages will need access to the same variables. Duplicating definitions is error prone, and so is duplicating boilerplate for hacking the shared definitions into every stage. The "base container" approach severely limits expressive power because it fixes the base container for every stage, while there are valid use cases where intermediate stages each need to use a different minimal base image that provides a tool required for the stage. |
@thaJeztah I have 11 base images that are only available internally and primarily used for our various builds. I am trying to make a base “FROM scratch” image, that those 11 images will use as part of a mulit-stage build, because some of the logic is the same across all 11, and this includes environment variables. So I have environment variables that need to be set within each image, and want to set these within my base image so that the same logic does not need to be replicated across every other image. |
@DMaxfield-BDS Scope of FROM scratch AS scratch-base
ENV foo=bar
ENV bar=baz build it, and push it to your (internal) registry, and those 11 base images could use it as base FROM scratch-base
RUN your stuff There is another proposal to have |
Thank you for your response. The one thing I did not point out is that these other 11 images all have different base images (python, npm, postgres, openjdk, etc). So what I am looking to do is put all the common set-up/prep into one base image, including setting needed environment variables used by my companies application, like the following:
Push to my internal registry. Then, an example of one of the other 11 images would do the following:
If I am able to use environment variables set in the 1st image, I don't then have to configure these same settings in each of the other 11 images. This allows me one configuration point, and allows better automation. |
How am I supposed to do something like this? RUN COMMIT_HASH=$(git rev-parse --short HEAD) How am I allowed to use In my opinion this design is very limited. Maybe one could: RUN COMMIT_HASH=$(git rev-parse --short HEAD) AS commit-hash |
@dnk8n this is what I do to work around it: On the build stage I do:
then on the other stage I copy the file with the data & and set the Environment variable before running my app that consumes it:
You should be able to do something similar to set the ARG. |
Using RUN is required to exit non-zero during both a `docker build` command with `--target` stage not defined, and with a `docker run` that uses an already-built image (See `Makefile` and `Dockerfile`) Also, move to a cleaner approach to sharing information from arguments across a multi-stage docker build, based on moby/moby#37345 (comment)
* fix(docker): Ensure docker run and build exit on stage failures Using RUN is required to exit non-zero during both a `docker build` command with `--target` stage not defined, and with a `docker run` that uses an already-built image (See `Makefile` and `Dockerfile`) Also, move to a cleaner approach to sharing information from arguments across a multi-stage docker build, based on moby/moby#37345 (comment)
* fix(docker): Ensure docker run and build exit on stage failures Using RUN is required to exit non-zero during both a `docker build` command with `--target` stage not defined, and with a `docker run` that uses an already-built image (See `Makefile` and `Dockerfile`) Also, move to a cleaner approach to sharing information from arguments across a multi-stage docker build, based on moby/moby#37345 (comment)
Using RUN is required to exit non-zero during both a `docker build` command with `--target` stage not defined, and with a `docker run` that uses an already-built image (See `Makefile` and `Dockerfile`) Also, move to a cleaner approach to sharing information from arguments across a multi-stage docker build, based on moby/moby#37345 (comment)
At the very least we should be able to do something like this to copy the ENV --from=stage1 FOO=$BAR This is even more important when considering using an external image as a stage because there can be important metadata in environment variables. ENV --from=nginx:latest FOO=$BAR |
I'm working-around by saving the environment to a file at each stage, using this pattern:
How that can finally be 'saved' into 'docker' such that it is visible when running |
Explicit way of copying (getting) environment variable from other stage is a must. I have a valid use case for it.
Now, how am I supposed to know the yarn version if it is defined in node image ( |
The multi-stage build is accommodated along huge benefits in our CI pipelines. A feature like "environment inheritance" would push it to another level maintenance and feature wise too. I'm having around 5 layers with heavy use of env vars and every update is a nightmare. The bright side(?), I rather think twice before I introduce a new stage. |
The current ARG implementation gives some curious behavior:
or
This is linked by the fact that ARGs are per-stage environment variables, not templating. I know that you (and me) have probably never see that in real life (string manipulation maybe ...). Why RUN don't interpret ARG before launching command like the legacy 'make' does?
|
24 hours are over =) |
Warning this solution doesnt work, although I wish it would! Write the env var to a file in the first stage and copy that one file in the second stage? etc? like this: FROM golang:1.14 as cm_base
ARG commit_id
RUN echo "$commit_id" > /tmp/env.json
FROM golang:1.14
COPY --from=cm_base "/tmp/env.json" "/tmp/env.json"
ENV cm_cc_commit_id="$(cat /tmp/env.json)" boom, except it doesnt work, since However what does work is writing to a file, making your entrypoint a bash file, and then doing this: #!/usr/bin/env bash
echo "the docker entrypoint args:" "$@"
export cm_cc_commit_id="$(cat /tmp/env.json)"
"$@" |
Think about it more like this: Dockerfile does not have a means of declaring variables except via global So yes, you need to explicitly import global It is definitely understood that the usage of global |
The reason for this was that ARG valus do not persist between build stages. moby/moby#37345 (comment)
The reason for this was that ARG values do not persist between build stages. More info: moby/moby#37345 (comment)
The reason for this was that ARG values do not persist between build stages. More info: moby/moby#37345 (comment)
The reason for this was that ARG values do not persist between build stages. More info: moby/moby#37345 (comment)
Docker ARGs are scoped per build stage (moby/moby#37345 (comment)). The `MIX_ENV` ARG is created in the `build` stage and is referenced in the `app` stage. This results in an incorrect `COPY` directory. The proposed change creates a global `MIX_ENV` ARG that is then usable in both stages.
In previous Docker versions, the `ENV` directive was scoped to the current build stage [1]. While this was relaxed by now so that "a stage inherits any environment variables that were set using ENV by its parent stage or any ancestor" [2], a stage still cannot expand variables that just were declared earlier in the file. The reason why this did not result in a build error is that unset variables are silently expanded to the quoted empty string ("") which matches everything. Fix that bug by hard-coding the paths. This also fixes another bug where copying from `sbtbuild` used the `DART_SDK` variable. [1]: moby/moby#37345 (comment) [2]: https://docs.docker.com/engine/reference/builder/#env Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
In previous Docker versions, the `ENV` directive was scoped to the current build stage [1]. While this was relaxed by now so that "a stage inherits any environment variables that were set using ENV by its parent stage or any ancestor" [2], a stage still cannot expand variables that just were declared earlier in the file. The reason why this did not result in a build error is that unset variables are silently expanded to the quoted empty string ("") which matches everything. Fix that bug by hard-coding the paths. This also fixes another bug where copying from `sbtbuild` used the `DART_SDK` variable. [1]: moby/moby#37345 (comment) [2]: https://docs.docker.com/engine/reference/builder/#env Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
In previous Docker versions, the `ENV` directive was scoped to the current build stage [1]. While this was relaxed by now so that "a stage inherits any environment variables that were set using ENV by its parent stage or any ancestor" [2], a stage still cannot expand variables that just were declared earlier in the file. The reason why this did not result in a build error is that unset variables are silently expanded to the quoted empty string ("") which matches everything. Fix that bug by hard-coding the paths. This also fixes another bug where copying from `sbtbuild` used the `DART_SDK` variable. [1]: moby/moby#37345 (comment) [2]: https://docs.docker.com/engine/reference/builder/#env Signed-off-by: Sebastian Schuberth <sschuberth@gmail.com>
Description
Afaik, there's no way to share variables between stages (correct me if i'm wrong, please). To share from one build stage to the next, the only option is to
COPY
files from one stage's directory to the current stage. We can build something like JSON files with a dump of build-time variables, but I think that this problem is a too frequent thing, and it will flood our multi-stage builds with all kinds of crazy JSON parsing issues. Can we automate this?Steps to reproduce the issue:
Describe the results you received:
Describe the results you expected:
Or maybe we can use something like namespaces to interpolate
${base1.v1}
The text was updated successfully, but these errors were encountered: