Infinite artifact re-deployment on a Windows 10 docker host

[MihaStemberger]

Hello!

I am having an issue that occurs only when running Docker on a Windows host machine. Artifacts I want to deploy to Wildfly never stop redeploying. The way deployment artifacts are passed to the container, is via a volume, that connects to /opt/jboss/wildfly/standalone/deployments.

docker-compose.yml file:

version: "3.7"
services:
  wildfly:
    image: "jboss/wildfly:20.0.1.Final"
    volumes:
      - ./_deployables:/opt/jboss/wildfly/standalone/deployments

I made an example project, that can be found here: https://github.com/MihaStemberger/docker.git

From what i have managed to debug, the issue is a wrongly set last modified timestamp because the last 3 digits of the timestamp round down thus making the timestamps different.

Here is an image of values while debugging wildfly-core-parent 12.0.3.Final version at FileSystemDeploymentService.java:1482 And this this is the stat result of the files that are made before and after the breakpoint.

sh-4.2$ cd /opt/jboss/wildfly/standalone/deployments/
sh-4.2$ ls -l
total 0
sh-4.2$ ls -l
total 4

---------------------
Before breakpoint
---------------------
-rwxrwxrwx 1 root  root  2494 Aug  7 13:25 wildfly_redeploy.war
-rw-r--r-- 1 jboss jboss   20 Aug  7 13:25 wildfly_redeploy.war.deployed
-rw-r--r-- 1 jboss jboss   20 Aug  7 13:25 wildfly_redeploy.war.isdeploying
sh-4.2$ stat wildfly_redeploy.war wildfly_redeploy.war.deployed  wildfly_redeploy.war.isdeploying

  File: 'wildfly_redeploy.war'
  Size: 2494            Blocks: 8          IO Block: 4096   regular file
Device: 33h/51d Inode: 1970324838626163  Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-08-07 13:25:31.793848400 +0000
Modify: 2021-08-07 13:25:31.786287800 +0000
Change: 2021-08-07 13:25:31.786287800 +0000
 Birth: -

  File: 'wildfly_redeploy.war.deployed'
  Size: 20              Blocks: 0          IO Block: 4096   regular file
Device: 33h/51d Inode: 1970324838626165  Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   jboss)   Gid: ( 1000/   jboss)
Access: 2021-08-07 13:25:35.867932900 +0000
Modify: 2021-08-07 13:25:35.559377700 +0000
Change: 2021-08-07 13:25:35.559377700 +0000
 Birth: -

  File: 'wildfly_redeploy.war.isdeploying'
  Size: 20              Blocks: 0          IO Block: 4096   regular file
Device: 33h/51d Inode: 1688849861915508  Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   jboss)   Gid: ( 1000/   jboss)
Access: 2021-08-07 13:25:35.867932900 +0000
Modify: 2021-08-07 13:25:33.107215700 +0000
Change: 2021-08-07 13:25:33.107215700 +0000
 Birth: -

---------------------
After breakpoint
---------------------
sh-4.2$ stat wildfly_redeploy.war wildfly_redeploy.war.deployed  wildfly_redeploy.war.isdeploying

  File: 'wildfly_redeploy.war'
  Size: 2494            Blocks: 8          IO Block: 4096   regular file
Device: 33h/51d Inode: 1970324838626163  Links: 1
Access: (0777/-rwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2021-08-07 13:25:31.793848400 +0000
Modify: 2021-08-07 13:25:31.786287800 +0000
Change: 2021-08-07 13:25:31.786287800 +0000
 Birth: -

  File: 'wildfly_redeploy.war.deployed'
  Size: 20              Blocks: 0          IO Block: 4096   regular file
Device: 33h/51d Inode: 1970324838626165  Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   jboss)   Gid: ( 1000/   jboss)
Access: 2021-08-07 13:25:35.000000000 +0000
Modify: 2021-08-07 13:25:31.000000000 +0000
Change: 2021-08-07 13:26:38.946329100 +0000
 Birth: -

  File: 'wildfly_redeploy.war.isdeploying'
  Size: 20              Blocks: 0          IO Block: 4096   regular file
Device: 33h/51d Inode: 1688849861915508  Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/   jboss)   Gid: ( 1000/   jboss)
Access: 2021-08-07 13:25:35.867932900 +0000
Modify: 2021-08-07 13:25:33.107215700 +0000
Change: 2021-08-07 13:25:33.107215700 +0000
 Birth: -
sh-4.2$

This causes FileSystemDeploymentService.java:865 to be true and invoke a re-deploy.

My docker info:

λ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  compose: Docker Compose (Docker Inc., v2.0.0-beta.6)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 3
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.72-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 12.38GiB
 Name: docker-desktop
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false

Any help would be greatly appreciated.

Edit1: this issue is also present if latest jboss/wildfly is used (as of Aug. 07.2021)

[tlf30]

I am having the same issue. If there is information I can provide to assist with fixing this, please let me know.

Edit: I am using WildFly 24.

[tlf30]

Just tested with WildFly 25, issue still exists.

[slominskir]

Issue is on Windows 11 too. With Docker Desktop 4.3.2 (72729) - currently the newest version available.

Also: Using WSL 2 based engine.

[tlf30]

I can confirm, I too can replicate the issue on Windows 10 and 11 with 4.3.2. I have not tested 4.4.2 yet.

[slominskir]

Same issue with Docker Desktop 4.4.3

[slominskir]

Probably worth mentioning Docker and Microsoft advise against mounting a Windows directory into a Linux Container using WSL 2. Neither go as far as to say it won't work, just that performance will be bad. See: https://docs.docker.com/desktop/windows/wsl/#best-practices. Using WSL 1 doesn't appear to be an option anymore on Windows 11 Home edition, so that leaves their other recommendation of installing projects in the WSL filesystem directly. This recommended approach involves a bit of indirection and assumes your IDE supports remote WSL directories (Microsoft Visual Studio Code and IntelliJ Idea both do). I've confirmed it does solve this problem though so is a reasonable, albeit annoying, workaround if your IDE supports it.

[davdr]

• I've encountered a similar problem. I believe the root cause is this WSL 2 issue: 9p client truncates modified time nanoseconds field to zero when setting it microsoft/WSL#7581 (it's confirmed but still open at the time of writing).
• You also be need to be running Java 9 or higher to be affected, because older versions of Java return file timestamps with second-precision only (details: https://stackoverflow.com/questions/24804618/get-file-mtime-with-millisecond-resolution-from-java). These older Java versions, as a side effect, thus make the deployment scanner's timestamp comparison "resilient" against the WSL 2 bug...