Saving file . for later use fails

[mrmartan]

Actual behavior

INFO[0029] Running: [/bin/sh -c npm install] 
{{...some NPM output ommited...}}
added 1635 packages from 1037 contributors in 15.068s
INFO[0044] Taking snapshot of full filesystem...        
INFO[0054] Pushing layer {{...redacted image repo...}}/cache:c0d37dfe3ddc014d09232c69bed53244ad6d2d61fb9d03162688c72de050574f to cache now 
INFO[0060] Saving file . for later use                  
error building image: could not save file: read /dev/agpgart: invalid argument

Expected behavior
This should simply work as the same Dockerfile used with docker works.

Step 3/34 : RUN npm install
{{...some NPM output ommited...}}
added 1635 packages from 1037 contributors in 16.028s
Removing intermediate container f98bb65dd64a
 ---> eac191bb4e90
Step 4/34 : FROM node:10.15.3 AS react-client-build-dev

To Reproduce
Running with gitlab-runner 13.3.2
Using Kubernetes executor with image gcr.io/kaniko-project/executor:debug-v1.2.0 ...
Steps to reproduce the behavior:

1. /kaniko/executor --cache=true --reproducible --context=$CI_PROJECT_DIR --dockerfile=$CI_PROJECT_DIR/Dockerfile --destination=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --cache-repo=$CI_REGISTRY_IMAGE/cache --snapshotMode=redo

Additional Information

FROM node:10.15.3 AS react-client-base
ADD ./client .
RUN npm install

FROM node:10.15.3 AS react-client-build-dev
COPY --from=react-client-base . .
RUN npm run build:dev

FROM node:10.15.3 AS react-client-build-tst
COPY --from=react-client-base . .
RUN npm run build:test

FROM node:10.15.3 AS react-client-build-prod
COPY --from=react-client-base . .
RUN npm run build:prod

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1 AS base
WORKDIR /app

ENV ASPNETCORE_URLS="http://*:80"
EXPOSE 80

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build
WORKDIR /server/src
ENV DOTNET_RUNNING_IN_CONTAINER true
ENV COMPlus_TieredCompilation 1
COPY . .
RUN dotnet restore ./server/src/MyApp.Server.Api -s https://api.nuget.org/v3/index.json -s http://nuget.someserver.com/nuget

FROM build AS publish
RUN dotnet publish ./server/src/MyApp.Server.Api --no-restore -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
COPY --from=react-client-build-dev /build /app/client-dev/build
COPY --from=react-client-build-dev /src/i18n /app/client-dev/build/i18n
COPY --from=react-client-build-tst /build /app/client-tst/build
COPY --from=react-client-build-tst /src/i18n /app/client-tst/build/i18n
COPY --from=react-client-build-prod /build /app/client-prod/build
COPY --from=react-client-build-prod /src/i18n /app/client-prod/build/i18n
ENTRYPOINT ["dotnet", "MyApp.Server.Api.dll"]

• gcr.io/kaniko-project/executor:debug-v1.2.0

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [No]
Please check if the build works in docker but not in kaniko	- [Yes]
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile	- [Yes]

[LouisStAmour]

See also: #960

[mattwilkinsonn]

Did you figure out how to solve this? I'm having the same issue.

[mrmartan]

I think I realized it is by design. Kaniko is not able to copy filesystem root over stages. I think it is in documentation. Although the error/feedback could use improvement.

[rkrell]

I have a similar issue with the latest gcr.io/kaniko-project/executor:debug from today:

INFO[0039] Taking snapshot of full filesystem...        
INFO[0042] Saving file . for later use                  
error building image: could not save file: copying file: read /proc/1/attr/exec: invalid argument

Is there an option how to work around this issue?

[puppetmaster]

I have a similar issue with the latest gcr.io/kaniko-project/executor:debug from today:

INFO[0039] Taking snapshot of full filesystem...        
INFO[0042] Saving file . for later use                  
error building image: could not save file: copying file: read /proc/1/attr/exec: invalid argument

Is there an option how to work around this issue?

We have the same issue here

[syaldram]

Hello,

I have similar issue as @puppetmaster mentioned above. Please advise.

[rkrell]

I have a similar issue with the latest gcr.io/kaniko-project/executor:debug from today:

INFO[0039] Taking snapshot of full filesystem...        
INFO[0042] Saving file . for later use                  
error building image: could not save file: copying file: read /proc/1/attr/exec: invalid argument

Is there an option how to work around this issue?

I solved this for my use case in the Dockerfile.
The problem has been in the scope of environment variables between stages.
For a DIND build on Gitlab, the following worked (cut down to the essential lines):

FROM python:3.11-slim as base

# Setup env
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONFAULTHANDLER 1


FROM base AS python-deps

ENV VIRTUAL_ENV=/opt/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

COPY ./requirements* ./

...


FROM base AS runtime

ENV APP_USER=appuser
# ENV VIRTUAL_ENV has been taken from the above stage
# ENV PATH has been taken from the above stage

...

COPY --from=python-deps $VIRTUAL_ENV $VIRTUAL_ENV

...

Thus, the effective COPY has been executed from /opt/venv in one stage python-deps to /opt/venv in the stage runtime.

In Kaniko I got the error described before, because $VIRTUAL_ENV has not been initialized in stage runtime (probably empty) and thus Kaniko copied the whole root filesystem from one stage to another, resulting in the permission errors.
I solved it by setting the environment variables in each stage separately, thus also in stage runtime:

ENV VIRTUAL_ENV=/opt/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

Here is the resulting essential part of the Dockerfile that fixed it:

FROM python:3.11-slim as base

# Setup env
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONFAULTHANDLER 1


FROM base AS python-deps

ENV VIRTUAL_ENV=/opt/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

COPY ./requirements* ./

...


FROM base AS runtime

ENV APP_USER=appuser
ENV VIRTUAL_ENV=/opt/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

...

COPY --from=python-deps $VIRTUAL_ENV $VIRTUAL_ENV

...

Thus, even if everybody here has a different use case, the common hint might be to prevent copying the root filesystem from one stage to another, because there is a lot of files in /dev, /proc etc. which just can't be copied by Docker COPY from one OS to another one. For me it was easy to find out, but for somebody else it might be trickier.

[kratsg]

@rkrell - I cannot believe I just stumbled upon this. You actually fixed my exact issue I was having where I had an ARG before the first FROM statement, and that was being cleared out even on the second one. I did it because I saw moby/moby#37345 (comment) that indicated it as a good solution for it... but it seems like kaniko doesn't support it which is not obvious.

I had also looked at #2203 and #1406 as well.

[rkrell]

@rkrell - I cannot believe I just stumbled upon this. You actually fixed my exact issue I was having where I had an ARG before the first FROM statement, and that was being cleared out even on the second one.
[...]

Actually, to make kaniko to be consistent with Docker: I'm not sure whether native dockerd supports it by mistake or by definition - in case the scope of ENV and ARG has to be a global scope from the line it has been defined over all following stages by definition it should be supported by kaniko as well. Or vice versa, if it is a bug in Docker it should not be adapted by kaniko :-)

Although the initially reported Dockerfile in this issue doesn't show this clearly and maybe the issue there is a different one. It looks like the path . is interpreted as / by kaniko, thus the root filesystem, and a not initialized ENV variable or ARG is replaced by / also by kaniko. Just guessing.