[Bug]: .NET 8 Container Permissions Changed Breaking Non-Root App User

[danielcrabtree]

Testcontainers version

3.8.0

Using the latest Testcontainers version?

Yes

Host OS

Windows

Host arch

x86

.NET version

8

Docker version

Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:18:56 2024
 OS/Arch:           windows/amd64
 Context:           default

Server: Docker Desktop 4.29.0 (145265)
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:01 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    26.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.13.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container. (Docker Inc.)
    Version:  0.0.27
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.23
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.4
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.6.3
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 26.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.133.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.31GiB
 Name: docker-desktop
 ID: 224a3cbc-8590-4017-89ab-50c7ac65bf25
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

What happened?

I'm using ImageFromDockerfileBuilder to build an image for an ASP.NET Core 8.0 project, then using ContainerBuilder to create a container for that image.

The container that gets produced by Testcontainers is different from the container produced by using docker directly to build the image then container.

Specifically, the files in the /app folder end up with different file permissions that are less permissive.

This causes the container to fail at startup with an error that ASP.NET cannot read appsettings.json due to lack of permissions.

.NET 8 has changed to build more secure containers, and it now uses a non-root user called app in the Dockerfile.

With the less permissive file permissions applied by Testcontainers, the non-root user app cannot read appsettings.json.

Changing the user back to root, as in earlier versions of .NET, resolves the problem.

Testcontainers shouldn't be producing files with different file permissions from the standard docker build as far as I can tell and there are no options to control this as far as I can tell. I haven't looked into it too deeply, but I saw mention in the docs that Testcontainers creates a tar of the files and then copies these in at some point. If so, I suspect that might be where the file permissions are being changed to less permissive values?

Relevant log output

No response

Additional information

No response

[HofmeisterAn]

Could you please share a reproducer? TBH, I do not entirely understand why the .NET version makes a difference. I am not aware of anything like this; our Weather Forecast example looks okay.

[danielcrabtree]

@HofmeisterAn Here is a minimum reproduction: WebApp.zip

---

Results from directly using docker with this project:

docker build . -t webapptest -f WebApp/Dockerfile
docker create webapptest --name webapptestcontainer

Container starts fine.

Files in /app have the following permissions:
appsettings are 755
other files are 644

---

Results from using Testcontainer

When using TestFixture in Tester to create the image and container using Testcontainers, specifically by running the UnitTest in debug mode with a breakpoint set on the exception thrown by container start (because the container fails to start). I had to do this, otherwise ryuk deletes the container before you get a chance to inspect it.

Container fails immediately with:
Unhandled exception. System.UnauthorizedAccessException: Access to the path '/app/appsettings.json' is denied.

Files in /app have the following permissions:
appsettings are 700
other files are 644

---

The issue appears to be that Testcontainers has somehow changed appsettings from 755 to 700.

In .NET 8 the Dockerfile uses non-root user "app". This breaking change in .NET 8 is documented here - https://learn.microsoft.com/en-us/dotnet/core/compatibility/containers/8.0/app-user

The Weather Forecast example in Testcontainers uses the "root" user. Changing the user to "root" is a workaround as "root" can access appsettings with permissions set to 700, but "app" user cannot. Note: I have not tested the Weather Forecast example specifically, but did check the Dockerfile while debugging this issue.

I am unsure why Testcontainers build is changing appsettings from 755 to 700. But that appears to be the underlying bug causing this issue.

[HofmeisterAn]

Thank you for the additional input. I now understand your point, which was not clear initially. I have not looked at it yet, but maybe there is an issue with how we inherit or set the permissions when creating the tarball.

testcontainers-dotnet/src/Testcontainers/Images/DockerfileArchive.cs

Lines 150 to 151 in 04262ed

	var entry = TarEntry.CreateTarEntry(relativeFilePath);
	entry.Size = inputStream.Length;

However, what I still do not understand at this point is how this could be an issue if the permissions are 700 (regardless of correctness). If the owner of the file and the process are the same user, it should not be a problem.

I cannot promise it. Today is busy for me, but perhaps I will have some time later to look at it. Thanks for bringing it to my attention.

Edit

FYI: If you break into this line, you will get the path to the intermediate tarball, where you can check its content and look at it further.

[danielcrabtree]

I went back to check and that is indeed another difference between the docker direct container and the Testcontainer container.

With docker direct, the owner and group for the /app folder is "app".
Whereas with Testcontainer, the owner and group for the /app folder is "root".
I'm not sure that this causing the current problem, but might cause issues elsewhere, i.e. if the application were to try and write anything to the /app folder.

In both docker direct and Testcontainer, the owner and group for all the files within the /app folder is "root". Which would be why there's no access despite being 700.

[danielcrabtree]

I checked the tarball and all the files in that are 700. So I assume that is where the problem is coming from, as the appsettings files are just copied across, whereas the other files with 644 are output by the build process. If this is the issue, to mirror the behavior of the direct docker build, the files in the tarball should be 755.

This doesn't explain why the /app folder has the wrong owner and group though. That would appear to be a separate bug?

[HofmeisterAn]

Ok, I think I can reproduce the issue. It appears that the .NET build / publish target simply copies the appsettings.json file and maintains its permissions, which is expected.

1. Building the image using the Docker CLI seems to use 755 as the default permissions:

docker build -f .\WebApp\Dockerfile -t webapp .
docker run -it --entrypoint /bin/bash webapp

app@7c402f65813c:/app$ whoami
app

app@7c402f65813c:/app$ ls -la
total 64
drwxr-xr-x 1 app  app   4096 May  7 19:03 .
drwxr-xr-x 1 root root  4096 May  7 19:03 ..
-rw-r--r-- 1 root root   991 May  7 19:03 WebApp.deps.json
-rw-r--r-- 1 root root  8704 May  7 19:03 WebApp.dll
-rw-r--r-- 1 root root 20716 May  7 19:03 WebApp.pdb
-rw-r--r-- 1 root root   469 May  7 19:03 WebApp.runtimeconfig.json
-rwxr-xr-x 1 root root   127 May  7 01:55 appsettings.Development.json
-rwxr-xr-x 1 root root   151 May  7 02:01 appsettings.json
-rw-r--r-- 1 root root   482 May  7 19:03 web.config

2. If we mount the directory from the host into the container and build / publish the project once again, we will see that it keeps the default permissions 777 (WSL):

docker run -it -v "$(PWD):/app" --entrypoint /bin/bash mcr.microsoft.com/dotnet/sdk:8.0

root@7410d217b4d7:/app# dotnet publish
MSBuild version 17.9.8+b34f75857 for .NET
  Determining projects to restore...
  All projects are up-to-date for restore.
  WebApp -> /app/bin/Release/net8.0/WebApp.dll
  WebApp -> /app/bin/Release/net8.0/publish/

root@7410d217b4d7:/app# ls -la /app/bin/Release/net8.0/publish/
total 116
drwxr-xr-x 1 root root   512 May  7 19:20 .
drwxr-xr-x 1 root root   512 May  7 19:20 ..
-rwxr-xr-x 1 root root 72520 May  7 19:19 WebApp
-rw-r--r-- 1 root root   991 May  7 19:19 WebApp.deps.json
-rw-r--r-- 1 root root  8704 May  7 19:19 WebApp.dll
-rw-r--r-- 1 root root 20708 May  7 19:19 WebApp.pdb
-rw-r--r-- 1 root root   469 May  7 19:19 WebApp.runtimeconfig.json
-rwxrwxrwx 1 root root   127 May  7 01:55 appsettings.Development.json
-rwxrwxrwx 1 root root   151 May  7 02:01 appsettings.json
-rw-r--r-- 1 root root   482 May  7 19:20 web.config

This doesn't explain why the /app folder has the wrong owner and group though. That would appear to be a separate bug?

As you can see in both examples, Docker behaves the same way. The USER instruction specifies the user that runs the process (RUN instruction).

If this is the issue, to mirror the behavior of the direct docker build, the files in the tarball should be 755.

Indeed, I think this is the issue. Setting the permissions (mode) while creating the tarball to entry.TarHeader.Mode = (int)Unix.FileMode755; seems to fix it. The test pass, and the files use the correct permissions.

However, I am not sure if we need this fix only for Windows hosts. IIRC, SharpZipLib inherits the permissions (mode) from the actual file. Which may not work properly between Windows and Linux (tarballs).

[danielcrabtree]

Indeed, I think this is the issue. Setting the permissions (mode) while creating the tarball to entry.TarHeader.Mode = (int)Unix.FileMode755; seems to fix it. The test pass, and the files use the correct permissions.

I've tried that change and can confirm it does fix the problem.

Regarding the USER instruction, in your tests above you have:

1. Building the image using the Docker CLI seems to use 755 as the default permissions:
   drwxr-xr-x 1 app app 4096 May 7 19:03 .

2. If we mount the directory from the host into the container and build / publish the project:
   drwxr-xr-x 1 root root 512 May 7 19:20 .

Note: that the app folder itself in 1 has owner and user of app, whereas in 2 it has owner and user of root.

When I run Testcontainers after applying the fix, I get:

$ ls -la
total 64
drwxr-xr-x 1 root root  4096 May  8 03:03 .
drwxr-xr-x 1 root root  4096 May  8 03:03 ..
-rw-r--r-- 1 root root   991 May  8 03:03 WebApp.deps.json
-rw-r--r-- 1 root root  8704 May  8 03:03 WebApp.dll
-rw-r--r-- 1 root root 20716 May  8 03:03 WebApp.pdb
-rw-r--r-- 1 root root   469 May  8 03:03 WebApp.runtimeconfig.json
-rwxr-xr-x 1 root root   127 May  8 03:02 appsettings.Development.json
-rwxr-xr-x 1 root root   151 May  8 03:02 appsettings.json
-rw-r--r-- 1 root root   482 May  8 03:03 web.config

Which corresponds to your 2nd test in regards to the /app folder owner and group. This difference as compared to building the image using the Docker CLI isn't currently causing me an issue, but I wonder if it might be a problem for other use cases? Is there a way to correct this difference?

However, I am not sure if we need this fix only for Windows hosts. IIRC, SharpZipLib inherits the permissions (mode) from the actual file. Which may not work properly between Windows and Linux (tarballs).

I'm just testing on my Windows dev machine with Docker Desktop currently and I haven't got a Linux docker server setup as of yet, so am unable to comment on this aspect unfortunately.

[HofmeisterAn]

Note: that the app folder itself in 1 has owner and user of app, whereas in 2 it has owner and user of root.

This difference as compared to building the image using the Docker CLI isn't currently causing me an issue, but I wonder if it might be a problem for other use cases?

I see. I need to double-check it with my other tests, but I do not expect this to be an issue.

I've tried that change and can confirm it does fix the problem.

I'm just testing on my Windows dev machine with Docker Desktop currently and I haven't got a Linux docker server setup as of yet, so am unable to comment on this aspect unfortunately.

For now, I would set the default permissions only for Windows hosts. If it is an issue with Linux hosts too, we can adjust it afterward as well. If I have some time, I can set up a test in Codespaces. It should not be difficult to reproduce there.

[danielcrabtree]

I need to double-check it with my other tests, but I do not expect this to be an issue.

The only situation I can think of would be writing to a new file in /app, e.g.:

File.AppendAllText("/app/test", "test");

This works in the Docker direct container, but fails in the Testcontainer container.

This isn't a problem for me as it feels like bad practice to store things in the /app folder, but I imagine someone might be doing it for some reason.

[HofmeisterAn]

This works in the Docker direct container, but fails in the Testcontainer container.

I am afraid, I do not know if we can even support that use case (although I agree that behaving the same way would be ideal). Testcontainers does not know anything about the USER statement. I am not sure how Docker does it either or why it is not working with the tarball. I mean, all we are doing is sending the tarball to the Docker engine. I suppose the Docker CLI contains some additional logic.

What I saw in the past is setting the expected permissions explicitly in the Docker build (Dockerfile), e.g.:

WORKDIR /app
RUN chown app:app /app
USER app

Maybe someone else has more insights or ideas on how to support this as well. Without taking more time and looking closer into it, I do not have any ideas for now.

Edit

The history looks pretty similar, except that the CLI build is using buildkit, as expected.

docker build -f .\WebApp\Dockerfile -t webapp . --no-cache --progress=plain

IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
0f668f2f6e43   6 seconds ago   ENTRYPOINT ["dotnet" "WebApp.dll"]              0B        buildkit.dockerfile.v0
<missing>      6 seconds ago   COPY /app/publish . # buildkit                  31.6kB    buildkit.dockerfile.v0
<missing>      21 hours ago    WORKDIR /app                                    0B        buildkit.dockerfile.v0
<missing>      21 hours ago    EXPOSE map[8080/tcp:{}]                         0B        buildkit.dockerfile.v0
<missing>      21 hours ago    WORKDIR /app                                    0B        buildkit.dockerfile.v0
<missing>      21 hours ago    USER app                                        0B        buildkit.dockerfile.v0
<missing>      2 weeks ago     COPY /shared/Microsoft.AspNetCore.App /usr/s…   24MB      buildkit.dockerfile.v0
<missing>      2 weeks ago     ENV ASPNET_VERSION=8.0.4                        0B        buildkit.dockerfile.v0
<missing>      2 weeks ago     RUN /bin/sh -c ln -s /usr/share/dotnet/dotne…   24B       buildkit.dockerfile.v0
<missing>      2 weeks ago     COPY /dotnet /usr/share/dotnet # buildkit       72.3MB    buildkit.dockerfile.v0
<missing>      2 weeks ago     ENV DOTNET_VERSION=8.0.4                        0B        buildkit.dockerfile.v0
<missing>      2 weeks ago     RUN /bin/sh -c groupadd         --gid=$APP_U…   8.46kB    buildkit.dockerfile.v0
<missing>      2 weeks ago     RUN /bin/sh -c apt-get update     && apt-get…   45.5MB    buildkit.dockerfile.v0
<missing>      2 weeks ago     ENV APP_UID=1654 ASPNETCORE_HTTP_PORTS=8080 …   0B        buildkit.dockerfile.v0
<missing>      2 weeks ago     /bin/sh -c #(nop)  CMD ["bash"]                 0B
<missing>      2 weeks ago     /bin/sh -c #(nop) ADD file:4b1be1de1a1e5aa60…   74.8MB

IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
0ef0c608b386   6 seconds ago    /bin/sh -c #(nop)  LABEL org.testcontainers.…   0B
eb727a51fd56   6 seconds ago    /bin/sh -c #(nop)  LABEL org.testcontainers.…   0B
c42ecfd00f55   6 seconds ago    /bin/sh -c #(nop)  LABEL org.testcontainers.…   0B
02a966e35ccd   6 seconds ago    /bin/sh -c #(nop)  LABEL org.testcontainers.…   0B
ecdfe2b0e6a6   6 seconds ago    /bin/sh -c #(nop)  LABEL org.testcontainers=…   0B
f893d3227446   6 seconds ago    /bin/sh -c #(nop)  ENTRYPOINT ["dotnet" "Web…   0B
9b709a195431   6 seconds ago    /bin/sh -c #(nop) COPY dir:f8c227044dede1532…   31.6kB
8adf83b1bad9   6 seconds ago    /bin/sh -c #(nop) WORKDIR /app                  0B
b3ddf35347cf   13 seconds ago   /bin/sh -c #(nop)  EXPOSE 8080                  0B
258fee5e9a48   13 seconds ago   /bin/sh -c #(nop) WORKDIR /app                  0B
0498c6485e69   13 seconds ago   /bin/sh -c #(nop)  USER app                     0B
4bf2fdf84219   2 weeks ago      COPY /shared/Microsoft.AspNetCore.App /usr/s…   24MB      buildkit.dockerfile.v0
<missing>      2 weeks ago      ENV ASPNET_VERSION=8.0.4                        0B        buildkit.dockerfile.v0
<missing>      2 weeks ago      RUN /bin/sh -c ln -s /usr/share/dotnet/dotne…   24B       buildkit.dockerfile.v0
<missing>      2 weeks ago      COPY /dotnet /usr/share/dotnet # buildkit       72.3MB    buildkit.dockerfile.v0
<missing>      2 weeks ago      ENV DOTNET_VERSION=8.0.4                        0B        buildkit.dockerfile.v0
<missing>      2 weeks ago      RUN /bin/sh -c groupadd         --gid=$APP_U…   8.46kB    buildkit.dockerfile.v0
<missing>      2 weeks ago      RUN /bin/sh -c apt-get update     && apt-get…   45.5MB    buildkit.dockerfile.v0
<missing>      2 weeks ago      ENV APP_UID=1654 ASPNETCORE_HTTP_PORTS=8080 …   0B        buildkit.dockerfile.v0
<missing>      2 weeks ago      /bin/sh -c #(nop)  CMD ["bash"]                 0B
<missing>      2 weeks ago      /bin/sh -c #(nop) ADD file:4b1be1de1a1e5aa60…   74.8MB

[danielcrabtree]

I think the workaround of setting the permissions explicitly when writes are required is satisfactory. Unlike the original issue that affects every project, this is a much more niche issue. I think documenting this difference and providing the workaround is adequate.

[HofmeisterAn]

I will create a PR that includes the discussed fix for Windows and updates the docs later in the day.