Proposal: Specify user/ownership for COPY.

[thaJeztah]

Proposal: Specify user/ownership for COPY

This is a docs-first proposal based on a discussion on #docker-maintainers (transcript up to here).

Background

The Dockerfile COPY instruction currently doesn't respect the user that is set using USER instruction; files and directories copied to the container are always added with UID and GID 0. Workarounds, for example, doing RUN chown ... don't work in all situations (for example when using aufs).

A previous PR, #9046, solved this problem by making COPY respect the user set via USER. Unfortunately, this would be a backwards incompatible change and was closed because of this.

The issue leading to that PR can be found here #7537.

Primary goals

• Enable setting ownership of files and directories copied to containers when using the COPY instruction.
• Maintain backward compatibility with existing Dockerfiles

Proposed change

Add a "configuration" option to the COPY instruction;

user=<uid_or_username> COPY <src>... <dest>

All files and directories copied to the container will be owned by the specified <uid_or_username>. After completing the COPY, the user that was active before the instruction will be used for subsequent instructions.

The configuration is optional. If no configuration is provided, files and directories are added with UID and GID of 0.

Note:
The discussion on #docker-maintainer does not mention specifying a group/GID.

Future plans

Although not explicitly mentioned in the discussion, "configuration" parameters can also be applied to other instructions in the future, for example, to set the user for a RUN instruction. Other "configuration" settings can be added as well.

This proposal is primarily for specifying the "user" for the COPY instruction. Still, future plans could be taken into consideration when commenting.

Known objections

A number of concerns were raised in response to the discussion on #docker-maintainers. I list them here, so that they don't have to be repeated in this discussion.

• The USER instruction is still available and may confuse users (when to use USER, when to use user=<uid_or_username>?).
• The proposed syntax makes the Dockerfile harder to read; instructions no longer at the start of each line.
• Handling of white-space if more "configuration" options are added in the future.

Process

To keep the discussion focused;

• The description of this proposal is also included as part of the PR. To comment on the description, create inline comments, this makes it easier to discuss things. The description on GitHub will be updated if commits are added.
• Please focus on the global "design" first. Comments on typos or language changes are welcome, but best kept for a later stage.
• I will update this PR by adding new commits to make tracking changes easier.
• If this proposal is accepted, I will hand the PR over to a maintainer for actual implementation. I'm not a Gopher 😄

[erikh]

/cc @tiborvass @shykes

[cyphar]

-1. This UX seems ... weird given the current UX of using single instructions to set metadata for other instructions (USER, ENV and WORKDIR).

[thaJeztah]

@cyphar thanks. To clarify, I'm personally -1 on the syntax, especially if USER and this proposal co-exist. I opened this PR (on suggestion of @erikh) to get the discussion started.

Something has to be done though, to fix the current situation, so suggesting alternatives that cover the mentioned goals is welcome.

[cyphar]

The only way that I can see it done is to depreciate the other instructions which will be replaced by this functionality. I still don't like the syntax all that much, but at least you won't have two conflicting UXes.

[duglin]

I would really prefer if we didn't introduce something that overlaps with the traditional setting of env vars, this will just lead to confusion for people. Its not clear to me why we don't just add support for options to our commands. For example:

COPY --user=foo src dest

The only issue I can see with this is that there might be an ambiguity between an option and the first parameter, but I think that can only happen on ADD and COPY so the risk is pretty small. And if that really is the only concern then I'm sure we can think of a way to address it - but first we need to decide on the overall direction.

[cyphar]

@duglin But they are sort-of like ENV values, in that they are metadata about the process which is not controlled by the process. Also, if we want to expand the syntax (see my point on consistency [this is becoming a running theme]), we should do it in a way that is future proof (what if we want to make user=root RUN be the way of doing what USER does now for RUN?).

[duglin]

@cyphar but they're not env vars, that's the point. I agree they might look like them but in the end they're not because they don't show up as env vars to the command (e.g. the cmd that RUN executes will not see them) and if they're not then we shouldn't confuse people by using the env var setting syntax.

I'm not sure I understand your 2nd point. Supporting options like normal commands do (via -x/--foo) is totally future proof.

I would turn your question around... what if we wanted to support the setting of "real" env vars on the RUN command, just for that one time? I would assert that
myvar=foo RUN ...
would be the way to do it.

Edit: granted we could support:
RUN -e myvar=foo ...
too but I don't think that's as natural for people, but it is an option.

[cyphar]

@duglin Sorry, I misread your example. I am okay with COPY --user=<> (I still don't like it though...).

[thaJeztah]

FYI, some alternatives were also suggested in the #docker-maintainers discussion see here. Perhaps I should add them to the description of this PR as well;

A: new instruction every time (COPYAS, COPYUSER)
B: pass config as a positional argument (COPY <user> <path>)
C: pass config as positional argument with KEY=VALUE syntax (COPY user=foo hardlinks=no)
D: prepend with a qualifier instruction, similar to ONBUILD. (USER foo COPY bar)
E: prepended with optional key-value config (user=foo hardlinks=no COPY...)
F: pass config as command-separated suffix (COPY,user=foo,hardlinks=no ....)

(slightly re-worded for better formatting)

[erikh]

Thanks for following through on this.

/cc @shykes

On Thu, 2015-01-08 at 08:14 -0800, Sebastiaan van Stijn wrote:

FYI, some alternatives were also suggested in the #docker-maintainers
discussion see here. Perhaps I should add them to the description of
this PR as well;

A: new instruction every time (COPYAS, COPYUSER)
B: pass config as a positional argument (COPY )
C: pass config as positional argument with KEY=VALUE syntax (COPY
user=foo hardlinks=no)
D: prepend with a qualifier instruction, similar to ONBUILD. (USER foo
COPY bar)
E: prepended with optional key-value config (user=foo hardlinks=no
COPY...)
F: pass config as command-separated suffix
(COPY,user=foo,hardlinks=no ....)

(slightly re-worded for better formatting)

—
Reply to this email directly or view it on GitHub.

[david-mccullars]

@thaJeztah, I'd like to suggest another alternative:

COPY name /path AS user

This could later be extended to work on RUN as well for consistency:

RUN cmd AS user

(The latter would help discourage frequent USER commands needed to flip back and forth between RUN's and promote the USER command as it was intended: a final instruction to specify the user that the container should run as.)

[ryneeverett]

+1 @david-mccullars. That provides a better combination of readability and consistency with the builder API than any of the previous suggestions.

[cyphar]

@ryneeverett @david-mccullars What if someone wants to use the word "AS" as part of the command line?

[pikeas]

For consistency with existing Docker syntax (CMD, ENTRYPOINT, possibly others), I propose extending COPY to accept an array argument similar to RUN:

COPY ["src", "dest", "user", "group"]

[thaJeztah]

What if someone wants to use the word "AS" as part of the command line?

I share @cyphar concern here; there's already a lot of code to handle ambiguous COPY instructions. If possible, we should try to avoid new problems. Also (although it isn't the primary goal here), I wonder how that notation would extend for other configuration-settings (perhaps USING user=foo other-config=bar?).

[ryneeverett]

@cyphar Good point. Syntax additions after RUN cmd might not be a good idea due to such ambiguities.

However, the critique isn't relevant to COPY and ADD. Is anyone even discussing special instructions for RUN though? My impression was that we were only looking to add them for COPY and ADD since (a)their implementation is particularly broken and (b)#9046 was rejected for the sake of "backwards compatibility".

[thaJeztah]

@ryneeverett unfortunately, it does apply; COPY source-path1 source-path2 dest is valid now, so COPY something AS destination would be problematic (AS being a path or instruction here?)

[ryneeverett]

@thaJeztah Ah, this is my first time seeing that in the docs. Option (B) COPY <user> <path> would also have to be rejected for this reason, since the <user> could be a <source>, right?

[thaJeztah]

Option (B) COPY would also have to be rejected for this reason

Yes. Looking back at the IRC logs, it also was rejected for that reason: IRC log

[duglin]

I still don't understand why we're looking for non-standard ways to pass options. What's wrong with:
COPY --user=foo src dest
why not start using -x and --xxx like normal commands do?

[duglin]

And, using -x allows us to easy support more options in the future. We've heard of people wanting to control whether archives are expanded vs just copied, as an example. Processing options based on order or special keywords will make that really messy. Let's not reinvent things.

[thaJeztah]

sgtm. Thanks everyone for taking part in this discussion and adding suggestions. You can subscribe to #10775 to check progress on the implementation.

[duglin]

Do we really want to close this? #10775 just adds the infrastructure to allow CMD --options, it doesn't actually implement any - like a --user for COPY.

[duglin]

@thaJeztah was this issue just to drive the options stuff or did someone actually want to set the owner during a COPY?

[thaJeztah]

@duglin hm you're right; closing this is probably a bit premature. This issue was created because #9046 was closed without an alternative and the "replacement" (options for instructions), wasn't really finalized. Yes, setting owner for COPY has been requested in various issues, so we need an implementation for that.

This should probably explain where we stand right now;

• Start a discussion on (alternative) syntaxes for per-instruction options
• Reach agreement on the syntax
• Create an implementation for the "options" framework/infrastructure (Add support for Dockerfile CMD options #10775) (in progress)
• Implement COPY --user=... using the new "options" infrastructure

I'm okay with closing this PR if we create a new "tracker" issue with the checkboxes above, wdyt?

[gfyrag]

Discussions seem closed here. Why about using a simple CHOWN/CHMOD instruction which will allow the builder to handle permissions on many architectures without side effect?
Flags on commands seems to be more complicated than necessary (just my opinion).
MARK/SQUASH seems good but does not fit some use case (Building image for different architecture - i build for arm on amd64).
Images are basically just file systems. Being able to manipulate this file system homogeneously and build image whatever architecture would be great.

[thaJeztah]

Why about using a simple CHOWN/CHMOD instruction which will allow the builder to handle permissions on many architectures without side effect?

@gfyrag because each instruction in a Dockerfile is executed in a new container, and thus a new layer in the layered filesystem. Adding CHOWN/CHMOD instruction to the Dockerfile syntax would be the same as;

ADD foo
RUN chown -r … chmod -r … 

[gfyrag]

I was thinking about a more low-level approach.
In my head, the builder handle CHOWN/CHMOD.

If i make an image for a different architecture than my laptop (arm for example), i can't do "RUN chmod xxx", i have a format error. But if the builder set permissions, i have no pb, the builder know the format of the files and can set proper permissions.

In fact, i think about this for any architecture, i write CHOWN/CHMOD but it can be SETOWNER/SETRIGHTS, working both on each linux architecture, or even on windows image (don't know how it will works).

[duglin]

See #6119 (comment) - a proposal is in the works.

[GKTheOne]

CHOWN/CHMOD (or whatever) will still leave the bloat issue. (pity, I like the idea, mainly because that's the only reason I want --user/--perm options on my COPY/ADD)

[etotheipi]

As far as I can tell, simply adding COPYUSER and ADDUSER directives solves this issue for 98% of the use cases. It avoids backwards compatibility issues, very straightforward to implement, and docs can simply suggest people use those forms instead of the original COPY and ADD.

Most importantly for me, it enables a pattern where the last USER statement in the base image is the default user in the derived image, and the developer doesn't even have to know about that user. Even if I accept the bloat of chowning/chmoding and the USER switching, I still have to know the username defined in the parent image:

USER root
RUN chown  ???:???  addedfiles/*
USER ???

The derived images should be agnostic to the username. This is not as flexible as some --user flag, but as I said it would solve 98% of use cases and encourage good behavior while also being nearly trivial to implement.

[duglin]

Dockerfile options (e.g. COPY --user=john ...) were specifically added to address this type of feature, but in the end the momentum just wasn't there to push it over the finish line and add their usage to the commands.

[ayen-tibco]

+1 Need to respect USER ownership in COPY and avoid the use of chown which bloats the entire image.

[wbednarski]

This is super confusing.

1. USER john # switch do user john
2. WORKER /home/john/stuff
3. COPY . /home/john/stuff # but copy as a root
4. RUN whoami -> john # root? no, again john
5. RUN ls -la -> root:root # all files are root's
6. RUN chown -R john:john /home/john/stuff -> Permission denied. # Are you kidding me?

[duglin]

See: #13600

[SharpEdgeMarshall]

@wbednarski yes, actually you have to do it like this

1. USER john # switch do user john
2. WORKER /home/john/stuff
3. COPY . /home/john/stuff # but copy as a root
4. RUN whoami -> john # root? no, again john
5. RUN ls -la -> root:root # all files are root's
6. USER root
7. RUN chown -R john:john /home/john/stuff -> now owner is john
8. USER john

[thaJeztah]

This feature has now been implemented through an optional --chown flag on ADD and COPY through pull request: #34263

The feature is included in Docker 17.09 and up, and allows you to do;

ADD --chown=someuser:somegroup /foo /bar
COPY --chown=someuser:somegroup /foo /bar

Or other combinations of user/group name (or ID);

--chown=someuser:123
--chown=anyuser:anygroup
--chown=1001:1002
--chown=333:agroupname

There are two additional feature requests in this area;

• Allow COPY command's --chown to be dynamically populated via ENV or ARG #35018 ("Allow COPY command's --chown to be dynamically populated via ENV or ARG")
• Proposal: Add --chmod flag to ADD/COPY commands (analogous to --chown) #34819 ("Proposal: Add --chmod flag to ADD/COPY commands (analogous to --chown)")

Given that the original problem was addressed, I'm locking the discussion on this issue to prevent it from collecting other (possibly unrelated) comments.

If you arrive on this issue because this feature is not working as expected, or you think other enhancements can be made; please open a new issue instead so that it can be tracked separately (feel free to add a link to this issue if you think it's relevant).