Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File mount does not update with changes from host #15793

Closed
greg0ire opened this issue Aug 24, 2015 · 69 comments
Closed

File mount does not update with changes from host #15793

greg0ire opened this issue Aug 24, 2015 · 69 comments

Comments

@greg0ire
Copy link

I have this nginx container, on which I'm mounting a vhost configuration file (indirectly, in fact, I'm using a data volume container). I would like to reload the configuration without stopping the container, with docker-compose kill -s HUP webserver, but when I change the configuration file on the host, it does not change on the guest, and vice versa. When I restart the container however, the guest takes into account the host changes. Here is an excerpt of my docker-compose.yml :

data:
    build: ./docker
    volumes:
        - .:/srv
        - ../symfony-1.4.20:/usr/lib/symfony-1.4
        - ./docker/conf/nginx_vhost.conf:/etc/nginx/sites-enabled/app.conf
        - .home-developer:/home/developer
        - $SSH_AUTH_SOCK:/tmp/agent.sock
webserver:
    image: greg0ire/nginx
    volumes_from:
        - data
    env_file:
        - ./docker-compose.env
    environment:
        - DNSDOCK_IMAGE=web
    links:
        - appserver
@cpuguy83
Copy link
Member

Can you provide output of docker info and docker version?
Have you inspected the data in the container after updating to verify?

@greg0ire
Copy link
Author

Containers: 33
Images: 207
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 273
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-26-generic
Operating System: Ubuntu 15.04
CPUs: 4
Total Memory: 3.85 GiB
Name: grosbill
ID: UFPX:PUSX:ZCTO:KZYC:VJD5:5R4A:32U7:H4QL:MHLZ:TBF3:J5GS:T3JB
WARNING: No swap limit support

uname -a
Linux grosbill 3.19.0-26-generic #28-Ubuntu SMP Tue Aug 11 14:16:32 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

Have you inspected the data in the container after updating to verify?

To check if the data was up-to date, I did this : docker exec -it carel_webserver_1 /bin/bash

I'm running all these containers in on a linux host.

Here is the Dockerfile for the nginx container. It

FROM ubuntu:trusty ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \
    apt-get install --yes nginx && \
    rm --recursive --force /var/lib/apt/lists/*
RUN ln --symbolic --force /dev/stdout /var/log/nginx/access.log
RUN ln --symbolic --force /dev/stderr /var/log/nginx/error.log
EXPOSE 80 443 ADD monitoring.conf /etc/nginx/conf.d/
ADD monitoring.html /usr/share/nginx/html/monitoring.html
ADD coverage.conf /etc/nginx/conf.d/
ADD dev.conf /etc/nginx/conf.d/
ENTRYPOINT ["nginx"]
CMD ["-g", "daemon off;"]

Here is the dockerfile for my data container :

FROM ubuntu:trusty

VOLUME /srv
VOLUME /etc/nginx/sites-enabled/app.conf
VOLUME /var/lib/mysql
VOLUME /home/developer/.composer

Not sure whether the VOLUME declaration in the data container is necessary BTW…

@cpuguy83
Copy link
Member

I bet I know what's happening here...

If you are using some editor like vim, when you save the file it does not save the file directly, rather it creates a new file and copies it into place.
This breaks the bind-mount, which is based on inode. Since saving the file effectively changes the inode, changes will not propagate into the container.
When the container is restarted the new inode.
If you edit the file in place you should see changes propagate.

This is a known limitation of file-mounts and is not fixable.

Does this accurately describe the issue?

@greg0ire
Copy link
Author

If you edit the file in place you should see changes propagate.

I just restarted my container, used nano to do the editing and it works! Thanks a lot. Feel free to close this, I am not sure if it should be considered as a bug or not.

@cpuguy83
Copy link
Member

Closing, it's not really a bug, just how linux works unfortunately.

@tcmal
Copy link

tcmal commented Nov 29, 2015

Is there any workaround for this? I don't think using nano is really viable in a project larger than 1 file. I'm using Sublime Text and I'd like to keep it that way.

@greg0ire
Copy link
Author

After reading this, I guess if I type set noswapfile, I might be able to continue using vim. Maybe Sublime Text has a similar option ?

@cpuguy83
Copy link
Member

@OscarOrSomething Mount a dir instead of a file.

@tcmal
Copy link

tcmal commented Nov 29, 2015

@cpuguy83 I am mounting a dir.

After some googling, Sublime text has atomic_save instead so Adding "atomic_save": false to user preferences worked (After a restart). Thanks @greg0ire

@greg0ire
Copy link
Author

I am mounting a dir.

Are you sure ? That's really strange… @cpuguy83 : maybe he is mounting a file and a dir (dunno if this is possible).

@tiaguinho
Copy link

@greg0ire I try use set noswapfile, but nothing change..
Isn't possible use vim with volumes?

@greg0ire
Copy link
Author

greg0ire commented Dec 1, 2015

Isn't possible use vim with volumes?

I don't know, but I'm interested in your results.

This breaks the bind-mount, which is based on inode. Since saving the file effectively changes the inode, changes will not propagate into the container.
When the container is restarted the new inode.

  1. Try finding out the inode of your file before, change your file with vim, and check that the inode changes. This should validate @cpuguy83 's theory
  2. If it turns out to be valid, then do the same, but set noswapfile before changing the file.
  3. Please report back with your findings.

@tiaguinho
Copy link

I made some tests @greg0ire, and this is the result.

With or without set noswapfile the inode changes, everytime the file is saved, but I made a little research how to make vim don't change the inode of a file.
The result was to use set backupcopy=yes, I try and the inode don't change when the file is modified.

Thank you guys.

@greg0ire
Copy link
Author

greg0ire commented Dec 2, 2015

And thank you for your feedback, it is interesting.

@tcmal
Copy link

tcmal commented Dec 5, 2015

This issue has come back for me a few days later. In both Sublime Text (3, with atomic save false) and Atom, The inode doesn't change at all. I assume this is an issue with the editors?

@tcmal
Copy link

tcmal commented Dec 5, 2015

It doesn't seem to be an editor issue (at least on atom)

The files do update but only after around a minute which is better but still incredibly annoying.

@thaJeztah
Copy link
Member

@OscarOrSomething I suspect that's because VirtualBox shared folders don't support inotify; https://www.virtualbox.org/ticket/10660

@tcmal
Copy link

tcmal commented Dec 5, 2015

@thaJeztah I'm not using Virtual box. Arch Linux Docker version 1.9.1, build a34a1d5-dirty

@thaJeztah
Copy link
Member

@OscarOrSomething oh, ignore my comment then 👍 (I know many people ran into that when using boot2docker/VirtualBox)

@sknuell
Copy link

sknuell commented Dec 14, 2015

Turning off sendfile off for Nginx solved this issue for me. Just put this in your vhost conf:

sendfile off;

The equivalent for Apache it is "EnableSendfile off".
See https://abitwiser.wordpress.com/2011/02/24/virtualbox-hates-sendfile/ for reference.

@greg0ire
Copy link
Author

@Sebwebdev : thanks, but I think you are a bit off topic : obviously, you are not mounting the file you are providing with your vhost one by one. The issue at hand is about file mounts.

@sknuell
Copy link

sknuell commented Dec 15, 2015

You are right, but @OscarOrSomething 's problem sounds exactly like my issue. Maybe it points someone in the right direction.

@greg0ire
Copy link
Author

I reread @OscarOrSomething and, after all he does not state if he was having the issue with the nginx configuration or with assets served by his webserver (but since he is having the issue with several files, the latter is more likely). So you might have the same problem indeed.

@OscarOrSomething : if you are having problems with assets, you should probably check the contents of the file from inside docker directly to know if the problems lies with docker or your webserver. The problem could be an HTTP caching problem if the webserver is at fault.

@tcmal
Copy link

tcmal commented Dec 15, 2015

Hi,

I checked and they are updating on the docker so it must be Apache.

Is this right?

/etc/apache2/sites-available/000-default.conf 

<VirtualHost *:80>
        # ...

        EnableSendfile Off
        EnableMMAP Off

        CustomLog ${APACHE_LOG_DIR}/access.log combined

        # ...
</VirtualHost>

@betoharres
Copy link

I'm facing this problem too, none of the suggestions worked for me. Any help? This kind of problem keeps me from developing in docker :( which I really want to

Im running docker machine in my OSX 10.11.4
My docker info:

Containers: 258
Images: 160
Server Version: 1.9.1
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 682
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 1.956 GiB
Name: default
ID: 624N:T5WM:NX7T:4RRP:GLAV:3TSL:OQXM:EEGM:WSUE:CSUL:AQQG:6UU6
Debug mode (server): true
 File Descriptors: 28
 Goroutines: 46
 System Time: 2016-03-16T18:43:53.655167222Z
 EventsListeners: 0
 Init SHA1:
 Init Path: /usr/local/bin/docker
 Docker Root Dir: /mnt/sda1/var/lib/docker
Username: betoharres
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox

docker version:

Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.5.1
 Git commit:   a34a1d5
 Built:        Sat Nov 21 00:49:19 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.3
 Git commit:   a34a1d5
 Built:        Fri Nov 20 17:56:04 UTC 2015
 OS/Arch:      linux/amd64

@cpuguy83
Copy link
Member

@betoharres Either don't use virtualbox, or disable sendfile in apache/nginx.

@betoharres
Copy link

@cpuguy83 thanks, disabling sendfile didn't worked at all, now the files sometimes changes and sometimes changes back to previous state when reloading the page without updating to the newest changes, urgh

@stcalica
Copy link

I just have a node express app running in docker using docker-compose volume to map my html files to the container. Getting the same behavior. It is not updating in the docker container. i did the same with vim and add set nobackup and other settings. Does anyone have any updates on this?

@moneal
Copy link

moneal commented Apr 14, 2016

I'm having the same issue with phpstorm and docker via virtualbox. The apache sendfile change didn't seem to help. I've also tried adjusting the save settings in phpstprm and no change.

@ZackKing
Copy link

have the same problem

volumes:
      - ./nginx/conf.d:/etc/nginx/conf.d

i change for this and it work

volumes:
      - /work/docker/nginx/conf.d:/etc/nginx/conf.d

i think the reason is

./

@digglife
Copy link

digglife commented Oct 16, 2020

Thanks to @greg0ire and @cpuguy83 . It's always a relief to me when some mysterious technical glitch that happened to me has a decent reason.

hemberger added a commit to hemberger/smr that referenced this issue Feb 15, 2021
Volume-mounting individual files is unreliable due to limitations
with the way Docker handles inode changes. If editing a file creates
a new inode, then those changes will unexpectedly not be propagated
into the container.

See moby/moby#15793 (comment).

To fix this, we will avoid volume-mounting files. We move the `.env`
file to `config/env`, so that we can reuse the `config` directory
volume-mount. We also create a symlink to `.env` for the convenience
of docker-compose commands (since it reads `.env` by default).
@santhosh-john
Copy link

If you edit the file in place you should see changes propagate.

I just restarted my container, used nano to do the editing and it works! Thanks a lot. Feel free to close this, I am not sure if it should be considered as a bug or not.

The nano approach is not working for me! :-(

@DiegoChajtur
Copy link

DiegoChajtur commented Jun 2, 2021

If you edit the file in place you should see changes propagate.

I just restarted my container, used nano to do the editing and it works! Thanks a lot. Feel free to close this, I am not sure if it should be considered as a bug or not.

The nano approach is not working for me! :-(

I edit a copy of file and after cat to new file:
vi file_new
cat file_new > mounted_file
this not break bind-mount
note: > overwrite entire file

@nicolajknudsen
Copy link

Ouch. So this is why we sometimes have to reload our containers inexplicably. This is a dreadful bug, I must say that I don't really agree with it being closed. When explicitly volume binding a file name I would expect changes to the file with that name to be reflected in the container without the need to have knowledge about the intricacies of the file system.

Is it really impossible to monitor for example the modified time stamp and update the inode reference accordingly?

@thaJeztah
Copy link
Member

It's how the linux kernel works 🤷‍♂️ you can try it on your host (on a linux machine, but probably works on macOS as well);

# create your file on the host
echo "# my-file" > myfile.txt

# create a directory (this is the "container's filesystem")
mkdir container

# create the mountpoint inside the container's filesystem
touch container/myfile.txt

# mount the file from the host into the container
mount -o ro,bind myfile.txt container/myfile.txt

# read the bind-mounted file "inside the container"
cat container/myfile.txt
# my-file

# "modify" the file, in this case, using `sed`, which uses a temporary file, then replaces the file
sed -i 's/my-file/my-file-updated/g' myfile.txt

# check that the file "on the host" is updated
cat myfile.txt
# my-file-updated

# check the mounted file "inside the container:
cat container/myfile.txt
# my-file

# unmount the file
umount container/myfile.txt

@videoMonkey
Copy link

This thread is amazing. I was running into this problem and effectively, yes, I was using vim and yes, set backupcopy=yes makes things work as hoped. Thanks for resolving my problem users of github 😊

@sre-data-engineer
Copy link

In Vim you can use backupcopy option set to yes

'backupcopy' 'bkc' string (Vi default for Unix: "yes", otherwise: "auto")

"yes" make a copy of the file and overwrite the original one

Making a copy and overwriting the original file:

  • Takes extra time to copy the file.
  • When the file has special attributes, is a (hard/symbolic) link or
    has a resource fork, all this is preserved.
  • When the file is a link the backup will have the name of the link, not of the real file.

@Krisslk
Copy link

Krisslk commented Dec 5, 2022

If you edit the file in place you should see changes propagate.

I just restarted my container, used nano to do the editing and it works! Thanks a lot. Feel free to close this, I am not sure if it should be considered as a bug or not.

hey, I am facing the same issue, did you start the docker container and used nano to edit the nginx.conf and save, I just tried it but didn't work, can you elaborate how you overcome this issue, big help, thanks

chibby0ne added a commit to chibby0ne/dotfiles that referenced this issue Dec 8, 2022
nvim:
* Use grubvox
* Create mapping to edit files in place when saving. (By default,
  vim/neovim create a new file when the opened file is saved and then
  replaces the opened file by the new one, effectively modifying the
  inode at every write).
This helps a lot when editing mounted files in docker using vim/neovim.
moby/moby#15793 (comment)

Signed-off-by: Antonio Gutierrez <chibby0ne@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests