docker cp to, from, and between containers #10198

jlhawn · 2015-01-19T21:12:54Z

please check the first commit with documentation changes for details

jlhawn · 2015-01-19T21:58:47Z

@SvenDowideit Please take a look at the updated docs commit 👍

thaJeztah · 2015-01-24T21:55:35Z

docs/sources/reference/commandline/cli.md

+The command will always fail if the `SRC_PATH` resource does not exist or if
+the parent directory of `DST_PATH` does not exist.
+
+> It is not possible to copy certain system files such as resources under


I think this should be formatted as a **note:**

Also, are there limitations wrt copying to/from volumes? If there are, they should be mentioned here (possible as a second/separate. "Note")

I think this should be formatted as a **note:**

Ah, thanks! I forgot that part from the style guide.

Also, are there limitations wrt copying to/from volumes? If there are, they should be mentioned here (possible as a second/separate. "Note")

Yeah, I think I included a part about this in the man page docs for it. I'll include it here too. I don't think the man articles are included on the website.

jlhawn · 2015-02-07T00:07:07Z

Closing for now until I get time to rebase and update.

Documented changes to API to enable new `docker cp` behavior. Added documentation on `docker cp` usage and behavior. Fixed up the docs/man/README.md file a bit. Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Adds TarResource and CopyTo functions to be used for creating archives for use with the new `docker cp` behavior. Adds multiple test cases for the CopyFrom and CopyTo functions in the pkg/archive package. Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Adds new jobs for handling uploading or downloading an archive of container content or copying files and directories between containers. Also adds ability to stat a resource in a container rootfs or mounted volume. To be used by the API for new endpoints making new `docker cp` behavior possible. Uses new pkg/archive utilities for archiving filesystem contents especially for this purpose. Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

duglin · 2015-02-12T15:13:48Z

api/client/copy.go

+
+// CmdCp handles copying files or directories to/from/across containers.
+func (cli *DockerCli) CmdCp(args ...string) error {
+	cmd := cli.Subcmd("cp", "[SRC_CONTAINER:]SRC_PATH [DST_CONTAINER:]DST_PATH", "Copy files between containers or a local path", true)


Note that this will fail on windows due to the colon. Sadly we can't even assume that a container name longer than a single char isn't a drive on windows because technically you can identify a container by just a single char.

Although, I don't know how else to solve it except to check the length of the container ID/name and assume that if its equal to 1 then its a drive letter. Not perfect but it should be pretty rare that people only use a single char to reference containers.

Will Windows Paths also use slashes, or back-slashes? i.e. a regular path on Windows will be C:\some\host\path. But, yes, this will become an issue if Windows containers are going to be supported I guess (I suspect that will still be far in the future)

This is a current issue that probably wont be fixed by this PR: #9932

Ah yes, I thought I remembered an issue. Guess this won't be the only location where docker will break on Windows, so not my biggest concern 😃

New endpoints have been added for copying to, from, and between containers. These endpoints call out to jobs registered by the daemon for handling each of these cases. Also added an endpoint to stat a filesystem resource in a container. Each handler uses a common error handling function for errors from these jobs before returning an error, if any. To be used by CLI for changes to `docker cp`. Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Utilizes new Remote API endpoints to handle copying to, from, and between containers in a manner that matches other common Unix utilities like `cp` and `scp`. Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Adds several integration tests for `docker cp` behavior with over a dozen tests for each of: container -> local local -> container container -> container Also adds several more integration tests with `docker cp` when volumes are involved in the paths which are being copied. Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Fixes moby#9787 Signed-off-by: Brian Goff <cpuguy83@gmail.com>

dmcgowan · 2015-04-03T23:20:41Z

+1 on design for adding to remote API

@tiborvass do PRs in design review need rebase before moving to code review?

@jlhawn can you add a TL;DR design description in the comment instead of just pointing to the documentation commit. The documentation commit is very detailed.

tiborvass · 2015-04-03T23:50:14Z

@dmcgowan you're right, they don't need a rebase if it's in design review. My bad.

~~From a Windows perspective, I guess we could discriminate the : separator by making sure it's never followed by a \, which would prohibit copying filenames that start with a \.~~

Also when specifying a file from a container, are we making sure the path is always absolute? I think we should do that, that way :/ means : is a separator between container name and path, and / is the start of the path within the container.

The idea is to prevent having docker cp C:\foo container:/bar mean copy file \foo from container C to file /bar in container container.

If we can agree on only allowing absolute paths for paths within containers, then the algorithm for detecting : separators is easy: : is a separator only if it's followed by a /.

Let me know if you guys agree with that at least.

thaJeztah · 2015-04-04T00:13:57Z

Will people try to run docker.exe inside Git Bash on Windows? (does that work?) I think that uses /c/Some/Path. Also, paths containing spaces are usually enclosed in quotes on windows (e.g. "C:\Program Files\").

dmcgowan · 2015-04-04T00:26:23Z

@thaJeztah is quoting the only option to escape spaces? I imagine the unix method of using a \ is not an option. It does raise a good point though, if we can rely on escaping or quoting for windows, it could be a non-issue docker cp "C:\foo container":/bar

thaJeztah · 2015-04-04T00:30:18Z

@dmcgowan hm, I'll have to check (escaping), but only run Windows at work, so not a Windows machine at hand right now

thaJeztah · 2015-04-04T00:32:25Z

Looks like a caret ^ can be used (weird choice); http://superuser.com/questions/279008/how-do-i-escape-spaces-in-command-line-in-windows-without-using-quotation-marks

tiborvass · 2015-04-04T03:50:48Z

@dmcgowan @thaJeztah don't forget that what counts from the code's perspective, is what gets passed to os.Args after the shell did its escaping. Also docker cp "C:\foo container":/bar would be invalid because there would only be one argument to cp (to be verified).

thaJeztah · 2015-04-04T08:48:36Z

don't forget that what counts from the code's perspective

Ah, true! The "Git Bash" console might still be something to take into account though.

tiborvass · 2015-04-21T00:30:45Z

I'd like to agree on having filepaths from containers always be absolute so that we can give the following definition: : is a separator only if it's followed by a /.

erikh · 2015-04-21T01:03:02Z

+1

duglin · 2015-04-24T17:29:02Z

I'm just wondering.... and I don't mean to derail things but.... if I'm remembering correctly this will not fully support scp-like action. For example, I don't think it support wildcards, do I have that right @jlhawn ?
So, if this is a precursor to supporting client-builds then I think we'll immediately get people complaining about all of the various limitations that it will have when compared with scp (wildcards, links, etc...). And then we're back to the (what feels like an endless discussion) around what to support and what to not support.

It got me thinking that it would be a lot easier to just support scp itself. This could mean two things:
1 - have an ssh server running someplace to be the receiver of the command
2 - get the same results via other (normal) unix commands.
For number 2, what I mean is:

Dockerfile:  COPY * /tmp/
would become:
tar -c * | docker exec -i *containerID* tar -x -C /tmp/

And it just works. All files, links, etc.... are copied into my container at /tmp.
We don't need to recreate normal unix/cp/scp/tar logic - we just need to convert COPY to tar.

Of course the hard part is that we need to make sure the target container is running and supports tar. But in reality we could just turn archive.go into tar if we really wanted.

Anyway, I guess what I'm wondering if whether we want to simplify this to just be add an API to "untar an incoming tarball into a container" (and we could of course, later look to do the reverse).

jlhawn · 2015-04-24T17:44:22Z

@duglin

I'm just wondering.... and I don't mean to derail things but.... if I'm remembering correctly this will not fully support scp-like action. For example, I don't think it support wildcards, do I have that right @jlhawn ?

Right. While your shell would do expansion on a local source file/dir, it doesn't support copying multiple sources to a destination directory (though we could certainly add this without much difficulty), and of course we have no way to auto complete sources that are in a container.

duglin · 2015-04-24T17:56:36Z

since we already have a 'cp' API to extract files as a tarball, I kind of like doing this slowly and adding the ability to put a tarball into a container. And, I wouldn't mind changing the 'cp' API to take wildcards :-)

But once we have those two APIs we can then look at what it takes to do the cp between containers. One concern with this current PR is that it will only copy between containers on the same host, right? Which means swarm won't be happy.

If we implemented (at first) "cp between containers" as the semantic equivalent of :
docker cp srcContainerID files | docker put targetContainerID dest
then, while there's a performance hit due to things coming back to the CLI, it will still work with swarm. It also allows the CLI to correctly deal with auth issues that might come up if people layer an auth-scheme on top of Docker, which I know some people have done. We can then explore optimizations to see if we can remove the CLI from the picture.

tiborvass · 2015-04-24T18:07:11Z

@jlhawn #10198 (comment) :)

jlhawn · 2015-04-24T18:20:57Z

One concern with this current PR is that it will only copy between containers on the same host, right? Which means swarm won't be happy.

I've talked to @vieux about this in the past and he mentioned that he kind of liked that it's a separate API endpoint so that they can implement it in a way that makes sense for swarm.

jlhawn · 2015-04-24T18:35:26Z

@tiborvass

I'd like to agree on having filepaths from containers always be absolute so that we can give the following definition: : is a separator only if it's followed by a /.

Is this in effort to reduce the ambiguity with Windows drive specifiers? like container_name:C:\foo\bar?
I'd like to be able to use source paths relative to a containers working directory and I'm pretty sure that we can solve this problem. Let me try to come up with a simple parser that handles all the cases (container path or local path?, absolute path or relative?). It looks like you and @dmcgowan have come up with an idea above.

tiborvass · 2015-04-24T20:08:58Z

@jlhawn how about we do that in another PR? It would just be a small convenient feature. We could move on with this PR then.

mgroff · 2015-04-29T17:48:46Z

+1 on moving forward with this PR!

LK4D4 · 2015-05-12T21:52:59Z

ping @docker/core-maintainers for design review

tiborvass · 2015-05-13T01:02:37Z

I agree with myself: #10198 (comment) and #10198 (comment) :)
and that we shouldn't do complicated things mentioned in #10198 (comment)

So +1 to that

jlhawn · 2015-05-13T01:30:52Z

I'm moving development of this feature to a new PR #13171 and closing this one. I'll also be reducing the feature set:

no longer accept paths relative to container workdir. All paths are relative to container root.
no longer pursue a copy-across feature/API endpoint.

These things can be added later.

The goal is: well-behaved copy operations that work more like cp -a.

@cpuguy83's PR #13125 may be a good alternative but it doesn't resolve the strange behavior where docker cp creates directories unexpectedly when copying only a file, copying directory when only the contents were desired, etc.

sidiki9 · 2019-01-17T11:32:09Z

hello, I just started with docker and I would like to know how to duplicate an application in a container to be able to depliyer it in another container?

thaJeztah · 2019-01-17T11:54:59Z

You'd normally build an image with the application, and start two containers from that image.

…

On 17 Jan 2019, at 12:32, sidiki9 ***@***.***> wrote: hello, I just started with docker and I would like to know how to duplicate an application in a container to be able to depliyer it in another container? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

sidiki9 · 2019-01-17T12:00:10Z

can you show me the procesuce? I am on ubuntu 18.04.1

jlhawn mentioned this pull request Jan 19, 2015

docker cp to, from, and between containers #9871

Closed

jlhawn force-pushed the new_docker_cp branch from 8e892a4 to 3b51500 Compare January 19, 2015 21:55

jlhawn added area/api /project/doc labels Jan 19, 2015

jlhawn force-pushed the new_docker_cp branch from 3b51500 to dab4c1f Compare January 19, 2015 22:07

thaJeztah reviewed Jan 24, 2015
View reviewed changes

crosbymichael added this to the 1.6.0 milestone Feb 6, 2015

jlhawn closed this Feb 7, 2015

jlhawn reopened this Feb 12, 2015

Josh Hawn added 2 commits February 11, 2015 19:35

Update docs for docker cp and its API changes

c7caafe

Documented changes to API to enable new `docker cp` behavior. Added documentation on `docker cp` usage and behavior. Fixed up the docs/man/README.md file a bit. Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

jlhawn force-pushed the new_docker_cp branch from dab4c1f to f9dfd54 Compare February 12, 2015 03:36

RoelVdP mentioned this pull request Feb 12, 2015

Moving a file from the host system to a container via docker cp #5846

Closed

SvenDowideit added the status/0-triage label Feb 12, 2015

jlhawn force-pushed the new_docker_cp branch from f9dfd54 to fd1a3b6 Compare February 12, 2015 04:22

duglin reviewed Feb 12, 2015
View reviewed changes

Josh Hawn and others added 4 commits February 12, 2015 10:46

api/client: New and improved docker cp command

d52db57

Utilizes new Remote API endpoints to handle copying to, from, and between containers in a manner that matches other common Unix utilities like `cp` and `scp`. Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

Fix matching for volume paths in cp

1d86684

Fixes moby#9787 Signed-off-by: Brian Goff <cpuguy83@gmail.com>

jlhawn force-pushed the new_docker_cp branch from fd1a3b6 to 1d86684 Compare February 12, 2015 18:47

icecrime removed the /project/doc label Mar 31, 2015

cpuguy83 mentioned this pull request Apr 24, 2015

Enhancement: Add support for COPY and ADD-like methods to the remote api #12757

Closed

cpuguy83 mentioned this pull request May 11, 2015

Allow copying files/dirs to a container - simpler implementation #13125

Closed

jlhawn mentioned this pull request May 13, 2015

docker cp to and from containers #13171

Merged

jlhawn closed this May 13, 2015

jlhawn deleted the new_docker_cp branch July 31, 2015 18:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker cp to, from, and between containers #10198

docker cp to, from, and between containers #10198

jlhawn commented Jan 19, 2015

jlhawn commented Jan 19, 2015

thaJeztah Jan 24, 2015

jlhawn Jan 26, 2015

jlhawn commented Feb 7, 2015

duglin Feb 12, 2015

duglin Feb 12, 2015

thaJeztah Feb 12, 2015

jlhawn Feb 12, 2015

thaJeztah Feb 12, 2015

dmcgowan commented Apr 3, 2015

tiborvass commented Apr 3, 2015

thaJeztah commented Apr 4, 2015

dmcgowan commented Apr 4, 2015

thaJeztah commented Apr 4, 2015

thaJeztah commented Apr 4, 2015

tiborvass commented Apr 4, 2015

thaJeztah commented Apr 4, 2015

tiborvass commented Apr 21, 2015

erikh commented Apr 21, 2015

duglin commented Apr 24, 2015

jlhawn commented Apr 24, 2015

duglin commented Apr 24, 2015

tiborvass commented Apr 24, 2015

jlhawn commented Apr 24, 2015

jlhawn commented Apr 24, 2015

tiborvass commented Apr 24, 2015

mgroff commented Apr 29, 2015

LK4D4 commented May 12, 2015

tiborvass commented May 13, 2015

jlhawn commented May 13, 2015

sidiki9 commented Jan 17, 2019

thaJeztah commented Jan 17, 2019 via email

sidiki9 commented Jan 17, 2019

docker cp to, from, and between containers #10198

docker cp to, from, and between containers #10198

Conversation

jlhawn commented Jan 19, 2015

jlhawn commented Jan 19, 2015

thaJeztah Jan 24, 2015

Choose a reason for hiding this comment

jlhawn Jan 26, 2015

Choose a reason for hiding this comment

jlhawn commented Feb 7, 2015

duglin Feb 12, 2015

Choose a reason for hiding this comment

duglin Feb 12, 2015

Choose a reason for hiding this comment

thaJeztah Feb 12, 2015

Choose a reason for hiding this comment

jlhawn Feb 12, 2015

Choose a reason for hiding this comment

thaJeztah Feb 12, 2015

Choose a reason for hiding this comment

dmcgowan commented Apr 3, 2015

tiborvass commented Apr 3, 2015

thaJeztah commented Apr 4, 2015

dmcgowan commented Apr 4, 2015

thaJeztah commented Apr 4, 2015

thaJeztah commented Apr 4, 2015

tiborvass commented Apr 4, 2015

thaJeztah commented Apr 4, 2015

tiborvass commented Apr 21, 2015

erikh commented Apr 21, 2015

duglin commented Apr 24, 2015

jlhawn commented Apr 24, 2015

duglin commented Apr 24, 2015

tiborvass commented Apr 24, 2015

jlhawn commented Apr 24, 2015

jlhawn commented Apr 24, 2015

tiborvass commented Apr 24, 2015

mgroff commented Apr 29, 2015

LK4D4 commented May 12, 2015

tiborvass commented May 13, 2015

jlhawn commented May 13, 2015

sidiki9 commented Jan 17, 2019

thaJeztah commented Jan 17, 2019 via email

sidiki9 commented Jan 17, 2019