Error copying a file containing an inferred ARG variable

[ferrastas]

Actual behavior
When trying to copy a file that contains a name build using an ARG variable, it fails with a "no such file or directory".

Expected behavior
To be totally honest docker also fails to build this image, but at least the message is more clear as we can observe that the filename cannot be constructed correctly. Meanwhile in kaniko it shows the correct path and filename, we spend hours debugging the previous stages and layers, clearly seeing the file that is referenced in the error.
The final clue has been Built cross stage deps: map[0:[/binaries/app_]] where we observed that weren't able to build the proper filename.

To Reproduce
I attached a simple Dockerfile and the kaniko and docker commands I used to test it.

Additional Information

• Dockerfile

FROM alpine AS builder

ARG VERSION=2.5.3

RUN mkdir -p /binaries \
  && echo "something" > /binaries/app_$VERSION

FROM alpine AS dist

COPY --from=builder /binaries/app_$VERSION /tmp/app

RUN cat /tmp/app

• Kankiko output

/kaniko/executor --no-push --dockerfile /code/Dockerfile
INFO[0000] Resolved base name alpine to builder
INFO[0000] Resolved base name alpine to dist
INFO[0000] Retrieving image manifest alpine
INFO[0000] Retrieving image alpine from registry index.docker.io
INFO[0001] Retrieving image manifest alpine
INFO[0001] Returning cached image manifest
INFO[0001] Built cross stage deps: map[0:[/binaries/app_]]
INFO[0001] Retrieving image manifest alpine
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
INFO[0001] Building stage 'alpine' [idx: '0', base-idx: '-1']
INFO[0001] Unpacking rootfs as cmd RUN mkdir -p /binaries   && echo "something" > /binaries/app_$VERSION requires it.
INFO[0001] ARG VERSION=2.5.3
INFO[0001] RUN mkdir -p /binaries   && echo "something" > /binaries/app_$VERSION
INFO[0001] Initializing snapshotter ...
INFO[0001] Taking snapshot of full filesystem...
INFO[0001] Cmd: /bin/sh
INFO[0001] Args: [-c mkdir -p /binaries   && echo "something" > /binaries/app_$VERSION]
INFO[0001] Running: [/bin/sh -c mkdir -p /binaries   && echo "something" > /binaries/app_$VERSION]
INFO[0001] Taking snapshot of full filesystem...
INFO[0001] Deleting filesystem...
INFO[0001] Retrieving image manifest alpine
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
INFO[0001] Building stage 'alpine' [idx: '1', base-idx: '-1']
INFO[0001] Unpacking rootfs as cmd COPY --from=builder /binaries/app_$VERSION /tmp/app requires it.
INFO[0001] COPY --from=builder /binaries/app_$VERSION /tmp/app
error building image: error building stage: failed to execute command: resolving src: failed to get fileinfo for /kaniko/0/binaries/app_2.5.3: lstat /kaniko/0/binaries/app_2.5.3: no such file or directory

• Docker build output

DOCKER_BUILDKIT=0 docker build -f Dockerfile .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  15.87kB
Step 1/7 : FROM alpine AS builder
 ---> c1aabb73d233
Step 2/7 : ARG VERSION=2.5.3
 ---> Using cache
 ---> b01f2869cfc9
Step 3/7 : RUN mkdir -p /binaries   && echo "something" > /binaries/app_$VERSION
 ---> Using cache
 ---> 72ec0ee0f403
Step 4/7 : FROM alpine AS dist
 ---> c1aabb73d233
Step 5/7 : ARG VERSION
 ---> Running in 5c084020cb0f
Removing intermediate container 5c084020cb0f
 ---> a09ce6273a01
Step 6/7 : COPY --from=builder /binaries/app_$VERSION /tmp/app
COPY failed: stat binaries/app_: file does not exist

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[braykov]

By definition, all ARGs are emptied after the first FROM clause is reached. If you want this to work, you need to set the ARG again after the second FROM in your Dockerfile.

[ferrastas]

You are probably right @braykov, but the message on Kaniko doesn't help to guess that, as the error shows the variable value interpreted.

error building image: error building stage: failed to execute command: resolving src: failed to get fileinfo for /kaniko/0/binaries/app_2.5.3: lstat /kaniko/0/binaries/app_2.5.3: no such file or directory

I did a quick test about the ARG propagation on a multistage build:

FROM alpine AS builder

ARG VERSION=2.5.3

FROM alpine AS dist

RUN echo "The value of VERSION is: $VERSION"

With kaniko:

/kaniko/executor --no-push --dockerfile /code/Dockerfile.2.5.patched
...
INFO[0001] Unpacking rootfs as cmd RUN echo "The value of VERSION is: $VERSION" requires it.
INFO[0002] RUN echo "The value of VERSION is: $VERSION"
INFO[0002] Initializing snapshotter ...
INFO[0002] Taking snapshot of full filesystem...
INFO[0002] Cmd: /bin/sh
INFO[0002] Args: [-c echo "The value of VERSION is: $VERSION"]
INFO[0002] Running: [/bin/sh -c echo "The value of VERSION is: $VERSION"]
The value of VERSION is: 2.5.3
...

With docker:

DOCKER_BUILDKIT=0 docker build -f Dockerfile.2.5.patched .
...
Step 4/4 : RUN echo "The value of VERSION is: $VERSION"
 ---> Running in 1d7b0ea12497
The value of VERSION is:
...

[ferrastas]

moby/moby#37345 (comment)

ARG VERSION=2.5.3

FROM alpine AS builder

FROM alpine AS dist

ARG VERSION
RUN echo "The value of VERSION is: $VERSION"

Kaniko:

INFO[0002] Args: [-c echo "The value of VERSION is: $VERSION"]
INFO[0002] Running: [/bin/sh -c echo "The value of VERSION is: $VERSION"]
The value of VERSION is: 2.5.3

Docker:

Step 5/5 : RUN echo "The value of VERSION is: $VERSION"
 ---> Running in 861560706a06
The value of VERSION is: 2.5.3

[ferrastas]

Dockerfile instructions, including ENV vars and ARG are scoped per build-stage, and will not be preserved in the next stage

This seems to be only partially true for Kaniko, we can use ARG values on other stages but it has a weird behavior when using the COPY --form as I initially shared on the issue.
Tough decision here 😓 or we change Kaniko to have the same behavior as Docker and introduce a regression or go full in, and add support for COPY