docker-compose copy file or directory to container

[ghost]

we miss a possibility to copy a file or directory using docker-compose. I find this really useful.
Please check many +1 in premature closed #2105

[shin-]

What's the usecase? Most of the suggested usage I've seen were antipatterns.

[ghost]

You can see some of many usecases clicking at link provided. As you can see many of subscribers consider it as really useful feature instead of "antipattern"

[ghost]

ooops, now I see "something" happened to issue #2105 as there are no comments at all anymore...
Perhaps I provided wrong link...

[ghost]

so, I find really useful to copy some configuration/initialization files to container. As example some *.sql stuff for db containers, some html/js/css content for apache/nginx containers or even jar file for java container. This will make it available/runnable "globally" not only on machine where it was composed as in case of mounting volume(s). Mainly this will be some combination of host-local and container-contained files. In fact any container can be considered useless without any configuration or initialization

this is correct link: #1664

[ukbea]

+1

[shin-]

This will make it available/runnable "globally" not only on machine where it was composed as in case of mounting volume(s)

The problem with this is that it is incredibly short-sighted (hence the term "anti-pattern"), as it will force you to repeat the operation every time the containers to be recreated. Not to mention the fact that it scales very poorly (what if you have 10 containers? 20? 100?)

The actual solution to your issue is to include those necessary files in your build (Dockerfile) and rebuild when an update is needed.

[ghost]

of course, if it is composed including all "shared" content into container, scaling 10-20-100- containers would be much easier. Everything you need is to pull it from repository and mount(yes, in this case mount) only node-specific config. And even more, you don't need run docker-compose on each node.
Sure we can use docker-compose in combination with build: & Dockerfile, however things become little more complex and yaml configuration in docker-compose is much more "elegant" :o)

[dkinzer]

I'm running into an issue where copy would come in handy (at least as an override). I mostly develop on mac so I almost never see an issue with commands running as root in the container and exporting to a mounted volume. However, recently using the same workflow on a CentOs has caused some major pain because files owned by the root user are being added to the host via the mounted volume. I would like in these cases to just be able to copy the host files to the container instead of mounting them.

The related issue: #1532

update

I think in my case I can get away with using COPY in the Dockerfile and having multiple docker-compose files one of which uses a volume mount.

[srghma]

Use-case:
I want to use directory from read-only file system inside container. Application creates new files in that directory, but because filesystem is read only this cause errors.

I can't use rw volume, because filesystem is read only.
I can't use ro volume, because effect will be the same.

It would be awesome to make writes that are persists only when container runs. I can make wrapper (https://stackoverflow.com/questions/36362233/can-a-dockerfile-extend-another-one) to only COPY files, but making this in compose, similar to volume, would be better

[nrdmn]

Use case: starting multiple docker containers simultaneously from .gitlab-ci.yml which need to write into the git repository's directory.

If the process inside a container fails or if the ci job is cancelled before the container has cleaned up after itself, the remaining files can't be deleted by gitlab-runner due to lack of permissions. Now I could copy the files within the container out of the volume into another directory, but that would be an antipattern, wouldn't it?

[harpratap]

Is this different from volumes: - ./folder_on_host/ :/folder_in_container/ ?
I am able to copy files from host to container (equivalent of COPY) this way in my compose file

[justin-vanwinkle]

@harpratap you are right, but the drawback is that /folder_in_container must not exist or must be empty or else it will be overwritten. If you have a bash script as your entry point, you could circumvent this by symlinking your files into the originally intended directory after you create a volume at /some_empty_location

+1 for having a COPY functionality. Our use case is for rapidly standing up local development environments and copying in configs for the dev settings.

[djelroy]

+1 for COPY. This would really be a helpful feature.

Use case: in swarm mode, I have a service using mysql image. I need to copy my initialization scripst in /docker-entrypoint-initdb.d/ so that MySQL can execute them.

Though it is possible to create an image on top of mysql, copy the files and use it or connect to the mysql
task in swarm and then manually run the scripts, it's kinda unnecessary in my opinion.

[AnishAnil]

+1 for COPY/ADD,

Use Case:
Fluentd requires the configuration files to be moved into the container during run time. These config files are created on the run time by our Jenkins Engine and without a COPY/ADD in docker compose it simply fails.

[jpz]

Suppose one has a shared config file across a number of docker machines, with their Dockerfiles in respective subdirectories under the docker-compose directory. How do you copy that shared config into each image? I can't symbolically link to ../ from the Dockerfile context without getting COPY failed: Forbidden path outside the build context

In this instance when running docker-compose build, I'd like to copy the config files from the docker-compose context prior to running the docker build steps.

I'm happy if someone can suggest a clean workaround of course.

[shin-]

Please don't comment with just +1 - it's a waste of everyone's time. If you have additional information to provide, please do so ; otherwise, just add a thumbs up to the original issue.

[Yournished]

What is the use of dogmatically insisting it is antipattern, just because in some cases it could eventually cause problems? This definitely has a good use as you could add one line to an existing file, instead of having to create an extra folder and file, then move the file to be added there. This pointless, bureaucratic creation of tiny files is the real antipattern, preventing users from creating simple and easy to maintain docker-compose files.

If users want to do harmful things with Docker, they will find a way no matter what you do. Refusing to add legitimate features just because someone may misuse them one day is foolish.

[markshust]

I created a quick gist for this. It assumes the docker compose service is named phpfpm, however you can change this to whatever you wish. feel free to modify.
https://gist.github.com/markoshust/15efb29aa5eebf8adae402af18b2e674

[sfuerte]

COPY/ADD would definitely be a welcome feature.

A usecase: running a Graylog instance in Docker for Dev purposes. In order to launch an input automatically, a JSON spec has to be put in /usr/share/graylog/data/contentpacks
With the COPY/ADD feature, it'll be as easy as single line in YML.

In order to get it working now (on Oct 16, 2018), need to mount a volume to that point AND copying the original content of that folder to the persistent volume. Which is quiet inconvenient.

[crazycodr]

I would benefit from that, i have a set of tools that import a database seed into a container and then i run the devtools database importer based on that file. I don't want to have to do:

docker cp "${seed_file}" $(docker-compose ps -q devtools):/tmp/seed_file

to be able to import my seed. And no, i will not compile my dev images with a fixed schema, this goes against web development pattern at the very least. Containers should be for app portability, not data.

It would make way more sense to do:

docker-compose cp "${seed_file}" devtools:/tmp/seed_file

All in all, it is just a short-hand that basically does the same thing, but it looks better to leverage docker-compose everywhere than to mix stuff...

[funkyfuture]

1. this seems to be a duplicate of Support `cp' sub command #3593
2. i agree with @shin- that the elaborated use-cases are following an anti-pattern
3. but wrapping up Docker's cp command makes sense, imo

[nrdmn]

@funkyfuture If you think that these use-cases follow an antipattern, then please suggest a solution that does not.

[psmolkin]

What about k8s-like "data section" ?
For example:

services:
  service1:
    image: image.name
    data:
      filename1.ini: |
        [foo]
        var1=val1
        [bar]
        var2=val2
      filename2.yml: |
        foo:
          bar: val1

or perhaps the same but for the volumes: section

volumes:
  service_config:
    data:
      filename1.ini: |
        [foo]
        var1=val1
        [bar]
        var2=val2

services:
  service1:
    image: image.name
    volumes:
      - service_config:/service/config

[Kreyren]

Any update on this?

I currently have a project which would benefit from this implemented as i have to use an ugly workaround to get directories from one container to another.

[peter-hartmann-emrsn]

Use case:

1. Run backup task within container via:
   docker-compose exec influxdb influx backup –database weather “/backup/weather.backup”
2. Get the backup file off the container e.g.:
   docker-compose cp influxdb:/backup/weather.backup ./weather.backup
3. Move the backup to another container, e.g.:
   docker-compose cp ./weather.backup influxdb:/backup/weather.backup

This then will be part of an admin script that helps moving backups between compose deployments. Such script then would look very clean. Mainly used during development and evaluation.

[hoemich]

I still wonder, why this should be an antipattern.
To me it would be just a solution to either:

1. Build new Images that crap your memory for simple changes to files that should be a "working copy"
2. Mount and have a cp be the first command in your entry point to create a working copy from the mounted volume to the real target.
   While the first solution is totally unusable as it requires a docker prune images multiple times a day, the second works, is fine but having it in the docker-compose would just #suckless

Just like the wait-for-it.sh script is fine, but having that possibility built into docker-compose would still be a great improvement in expressivness.

[viafcccy]

Can add copy!
When container start can copy some file to host machine!

[ndeloof]

Closing this as implemented in Compose v2

[daniel-shuy]

Yay! Any example of how the syntax would look like?

[ndeloof]

same as docker cp but targetting services:

 docker compose cp SERVICE:SRC_PATH  DEST_PATH

[daniel-shuy]

Nice! Will there also be a way to define this in the Docker Compose file?

[robclancy]

compose officially an anti-pattern enabler, thank god.

[ndeloof]

@robclancy the anti-pattern is the way some might use this command.
Could say the same about docker run --privileged

We introduced this command as docker cp exists for a while, so there no reason docker compose does also support this.

[evbo]

@ndeloof in docker cp you are copying data to/from a running container. Is the same true for docker-compose cp? Or does it persist the change to an image?

I'm hoping the latter is true, and that this can be configured inside the docker-compose.yaml file?

And what version of docker-compose is this supported under?

[ndeloof]

@evbo docker compose cp does the same as docker cp for service's container(s):
it copy file into a running container. Nothing committed/persisted. This command was requested by many, and as an existing docker one I don't see any reason compose doesn't offer the same, with the same limitations (non persistent changes)

Also, this is command line only, there's no equivalent in compose file. Declaring a volume bind sounds a reasonable alternative if you want this set in your yaml configuration.

this is implemented in Docker Compose v2, not docker-compose

[evbo]

ok, thanks for clarifying @ndeloof

If I understand correctly, the main benefit is that docker-compose cp recognizes services (not just containers), so does that mean if I have a scale > 1 it would cp the data to each and every instance?

What if I have a scale: 0? No cp happens? Are there other benefits over docker cp?

[ndeloof]

this is indeed "just" docker cp applied to services
for scale != 1 it will copy to all containers, or you can pass --index

[natotech]

What's the usecase? Most of the suggested usage I've seen were antipatterns.

One use case for me is to put the file custom.cnf into /etc/mysql/conf.d inside mysql container. Volume is not an answer to that but file copy would be more appropriate. So you expect me to create a Dockerfile just copy that one file?

[jadon1979]

Common Scenario: 1 or more files that live in non-empty directories need to be changed on-the-fly. You can't use volume with these directories as you'd lose all the other files in the directory. You most definitely do not want to copy the entire directory down to work around this. A suggestion of rebuilding the image every time you make a change completely ignores the real world, time, and the rage induced because you missed a comma. A basic copy command isn't an anti-pattern by any stretch.

copy:
  - ./some_dir/some_file:/etc/app/some_config/:<chmod>:<chgrp>:<chown>
  

A quick implementation of this might be an underlying volume that takes these files and creates a symlink to their respective destinations.

[ghost]

I just want to add to what @jadon1979 said, that any and all workarounds to the copy conundrum suffer in one of many ways:

• docker COPY command requires you own the image you're trying to copy into, else you need to create yet another image
• docker context (that you COPY from), requires everything to be co-located, typically forcing you to set the context to a parent directory in order to reach various disparate resources you may need.
• but as the context gets set to a higher and higher parent directory, the likelihood of unwanted files grows and so you must maintain a .dockerignore file, which can get pretty annoying especially when your cache is breaking because you forgot to ignore something new added to your context
• Then there's volumes, but their bind mount chowns everything to root, completely breaking a developer's access

There was recently a PR merged for docker-compose v2 that performs a docker cp. @shin- Just thinking outside the box here, but what if something like @jadon1979 compose syntax above supported this and each time you build the container it performs the cp for you? Could even extend this further to cp something out of the container after build too (very useful if you're using docker only for compiling a binary, etc.)!

[eldarj]

why not add this to docker-compose and enable for use in docker-compose.yml configs as well?

[XeJames]

I solved it. Gave up on Docker entirely and am using LXC. Solved everything for my use case AND no more people insulting me for "antipatterns". Win-win.

[alamar]

It is disingenous to write that the feature was implemented, because it was not.

The feature that was asked for have not been not implemented, and instead the feature that nobody has asked for and which was already fully covered by underlying docker was added.

The central issue of this feature request is that one does not own most of the images they have in their Docker Compose file, and when they bring up an image like grafana/grafana they cannot supply its config file in a hygienic fashion, and the container would not boot without that file making subsequent docker cp (or docker compose cp for all what it is worth) impossible. Bind mount of that file will lead to the config file hanging around in the host file system, which does seem like a huge anti-pattern WRT docker to me.

[washtubs]

compose-spec/compose-spec#346

This ends the nearly 6 year long holy war. The reasonable among us were just asking for a way to put configs in our containers without volume mounts and without rolling new images.

Apparently it's available in docker compose 2.23.1. I would follow that thread if you are still seriously interested in this feature.

[ndeloof]

@washtubs while inlined config indeed don't rely on bind mount, this is not the reason this feature was introduced. We could support file-based configs without a bind mount, just this would have some side effects on existing users.

@alamar the example you mention suggest we should investigate lifecycle hooks. A pre-start hook to cp required file in container would solve this use-case.

[washtubs]

@ndeloof I haven't used docker compose in some time. I got renewed interest in it just now after realizing it was rewritten in go (yeah it's been a while), and seeing the output of docker compose up which honestly looks gorgeous, which is why I got curious to see if anything became of this issue.

So forgive me for my lack of ecosystem knowledge here. I assume those who are used to helm and kubernetes would like to have some kind of templating pre-processor engine on top of docker compose and use that to insert file contents inline. I'm not sure if compose has a "helm"-like sister software like kubernetes does, or perhaps even builtin support. But that's an approach that would be familiar to me which I would leverage to insert config files inline as a pre-processing step.

[ndeloof]

@washtubs docker compose do support variable interpolation to offer some flexibility, but we don't want this to become a templating system like the one helm uses. Obviously someone could built another tool to produce compose.yaml from another source.

[andoks]

@washtubs / @ndeloof : in a way docker compose already can be a lightweight templating tool for itself, by building your docker-compose config with variables, then "rendering" the complete result by using docker compose config 😅

[washtubs]

That doesn't seem quite the same. I think that is basically coalescing multiple yaml files together into one, another thing that helm does. But helm uses go's template engine which gives a ton of flexibility on the kinds of things you can insert, and manipulations you can do, like quoting. A common pattern is inserting the contents of a file inline with indent. You can even define your own custom template functions to be reused throughout your project.

Honestly you could probably use helm to do this for compose, just make a helm chart alongside your docker-compose file, and you can use the helm template command to get the output docker-compose file with your file contents rendered and script the deployment yourself. (won't be able to use helm install or upgrade obviously, since those are k8 specific)

[sahlex]

Another usecase: I have a compose file on a local machine and my dockerhost ist a remote machine. How do I reference a file to be mounted from my local directory? Answer: not at all without building a specific image! A solution would be to copy it into the container.