Add support for specifying .dockerignore file with -i/--ignore

[tristanz]

As several people have mentioned (@thaJeztah, @duglin) in #9707, it would be great to be able specify the .dockerignore file using -i/--ignore in conjunction with named dockerfiles. It is often difficult to use named dockerfiles because the build context becomes too large.

[runcom]

#dibs or sorry @tristanz are you already working on it?

[duglin]

To be clear, I didn't say I wanted it - just that it could be supported.
You may want to see if there's buy-in from the team first before you implement it.

[duglin]

@runcom ^^^^

[runcom]

roger that!

[tristanz]

I'm not working on about it.

It seems like a natural option given -f, since -f makes it much harder to use .dockerignore effectively. Without -i it is impossible to use -f for our repo because our build context is 1GB+. We have pretty classic setup: single repo with many services that share code. We'd like to be able to build all these services without custom scripting or slow builds.

[thaJeztah]

Not a maintainer (for this), I'd be +1 for this so that different Dockerfiles can exclude different parts of the build-context.

I'm not sure having a single-char flag (-i) is needed, perhaps --ignore-file (to match, e.g. --env-file).

@tristanz Perhaps you can describe some use-cases for this feature, it may help the maintainers make a well-founded decision whether or not this is something that's wanted

edit:
Our comments just "crossed", I see you just added some description on your use-case

[tristanz]

Yes, I believe our use case is exactly the use case described in the named dockerfile thread: a single repo with many services that share libraries. So this is just trying to make that same use case work for real setups. Single repos like this tend to be large (which we love for many reasons!), so without --ignore-file we can't use -f and are forced to resort to prebuild scripts.

[duglin]

@tristanz can you give a little more detail on these large repos? In particular, when I think about large repos with multiple sub-projects, I think to think of it as each project having their own sub-dir with their own Dockerfile but they all share a common dir that might be a sibling to each project. For example:

/common
/project1
/project2

and in these cases I wonder if supporting symbolic links wouldn't be easier. Meaning, /project1/common is a symlink to /common.

Also, to be clear, I'm not for or against a --ignore flag - still trying to understand the usecases.

[tristanz]

@duglin that's my setup exactly. I actually would much prefer symlink support to solve this. It feels less docker specific, allows be to leave my dockerfiles unchanged (sub folder build context), and I can avoid writing lots of dockerignore files. I was just under the impression that named dockerfiles were the recommended alternative. I see symlinks as the better alternative by far.

[thaJeztah]

Are symlinks supported on Windows nowadays? (Wondering)

For my personal use-case, some containers are created using a volume to mount the source-code during development. Being able to exclude those files (huge number of files, not the size per-se) when starting the dev container will save lots of time. (And, yup, there are workarounds)

[tristanz]

I guess the main question is whether the really is no solution for multiple container git repos with shared code. This seems like a extremely common use case, but any discussion of symlinks or relative paths is not finding traction as far as I can tell.

Here's the folder structure I'm thinking about:

/docker-compose.yml
/container1
   - Dockerfile
/container2
   - Dockerfile
/shared

Is there really no docker way? The best I have found so far is to simply copy shared into multiple places.

[tristanz]

Is it worth working on a pull request for this feature?

[thaJeztah]

Is it worth working on a pull request for this feature?

It hasn't been decided on, yet, but you could try. With the chance it gets turned down :)

Sometimes discussing a PR helps making a decision, so..up to you, I guess :)

[Ralle]

Another solution would be to look for the .dockerignore file where the Dockerfile is stored as the build directory is not necessarily the directory of which the Dockerfile is, but I would also prefer -i.

[thaJeztah]

Another solution would be to look for the .dockerignore file where the Dockerfile is stored

Unfortunately, that won't work if multiple Dockerfiles are in the same directory (e.g. Dockerfile.dev and Dockerfile.prod)

[Ralle]

Right. Then how about a Dockerfile command to describe the corresponding .dockerignore?

[thaJeztah]

I think the proposed solution (a --ignore-file option) is the best approach, that's most flexible. Only thing needed now is to decide if this is something that is desirable enough to implement.

[tristanz]

There are two thing I dislike about named dockerfiles even with dockerignore.

1. Dockerignore becomes needed everywhere.
2. Lots of containers don't have shared dependencies, but it's not clear to users of my repository what the build context should be. It would be fine if the root context was used everywhere, but our repository has forks of public Dockefiles that assume local context only.

Dereferencing symlinks seems much more elegant. I can always see all dependencies in each folder, like single Dockerfile repositories. Security could be handled not allowing symlinks to escape the outside the local folder unless an environmental variable or flag is set.

[tristanz]

I take back my second point. A better convention would be to put Dockerfiles at the depth of the build context. So Dockerfile.web rather than web/Dockerfile. This would make the context clear enough.

In terms of this whole issue, is there a reason why the build context is the entire folder not simply files and folders that are explicitly ADDed (plus Dockerfile/.dockerignore)? This would make multiple .dockerignore mostly unnecessary.

Dereferencing symlinks also looks like a non-starter because there is already a defined behavior -- create the symlink rather than dereference it.

[Ralle]

I make a web project and have billions of files. Having ADD for each would create an immense amount of intermediate containers and I would need a script that built my Dockerfile for me.

[tristanz]

@Ralle I don't think anything would change. You just add a folder like normal. I'm just suggesting avoiding tarring and uploading things you never add.

[pikeas]

+1 to, in order of preference:

1. Build context shouldn't send all files, as described by @tristanz above. To maintain backwards compatibility, the CLI flag could be --full-context defaulting to true.
2. Symlinks followed during build.
3. Custom .dockerignore with -i per this issue.

[tristanz]

I think 1) in @pikeas list is by far the most elegant, but it is currently not easy to implement since parent images may have ONBUILD instructions.

[himmetozcan]

Here

Here it says, when the BuildKit backend is used, build looks for dockerignore file that is relative to the dockerfile path, so if there are multiple dockerfiles specified in the build, it will look for multiple dockerignore files, right?

[Mugane]

Maintaining multiple Dockerfiles for different contexts is absurd, we have one that generates and runs different bash scripts depending on build target to do the necessary container setup. Other than being absolutely ridiculous that there is no conditional capability in a Dockerfile, it is equally absurd that the only way to specify different .dockerignore files based on target in a docker-compose command is to generate custom files on-the-fly. The assumption that each build target would have a separate Dockerfile is overly presumptuous (there are plenty of completely reasonable configurations that would only need to have a multiple Dockerfile setup solely to address this need for optimizing production targets), and is based on a tooling strategy that is fundamentally flawed (gross reuse of code, a maintainability nightmare). Even if this were not the case, nobody has any business whatsoever forcing this convoluted methodology on the world, and in my opinion it is yet another fail for a technology that was supposed to make it easier to deploy apps across platforms and instead created a whole industry where disillusioned Frankenstein coders have to hack together ridiculously unmaintainable messes to get anything done.

I don't understand the motivation for holding back on adding a -di parameter for docker-compose or even better allowing the .dockerignore path/filename to be specified in the docker-compose.yml file. Why was this not done to start with and why is it even up for discussion?

[dm17]

@Mugane Agreed, and it makes a lot of sense to consolidate complexity on the docker-compose level, rather than swarm (or even more complex) level. That means, all complexity in this thread and docker/compose#3729 should be doable in compose - and generally compose is a good place to consolidate arising complexity.

[piotrekkr]

Forcing people to use multiple (basically same) dockerfiles just so they can load different .dockerignore files is PITA.

My situation is like this:

• I'm using multi stage build for my image, (base stage, then prod and ci stages that are created from base stage)
• CI image should contain some test and tools configuration files
• prod image should not contain any such files

In such situation it seems normal to have something like this:

.dockerfile - standard excludes for all stages
.dockerfile.prod - additional excludes for production

and when building prod image I would just specify --ignore-file .dockerignore --ignore-file .dockerfile.prod and that is it.

Someone already proposed such solution in #29405 but it was discussed in the maintainers meeting and somehow this is not the right approach

Somehow better approach is to:

• copy Dockerfile with different name like Dockerfile.prod
• copy paste .dockerignore file and name it Dockerfile.prod.dockerignore and add additional ignore lines

Now we need to maintain two Dockerfiles and two ignore files instead of one...

I just cannot get how this is better approach.

[dhirschfeld]

in the description and tests you will see how the file/s should be named and where to be placed.

The tests don't exhaustively test all requested combinations. In particular they say nothing about how to use this feature for dockerfiles that have meaningful names and extensions.

After wasting my time exhaustively testing all combinations of naming and file placement it turns out that if you have a dockerfile ./app-folder/myapp.docker then the dockerignore file has to be named ./app-folder/myapp.docker.dockerignore - i.e. the .dockerignore extension gets added to the original extension rather than replacing it.

[dtheodor]

This is unbelievable. Can't you just support passing a dockerignore and let users move on with their lives instead of this god awful file name hack? At the moment I am resorting to piping tar output to docker build because the whole configuration of what to include/ignore is a complete mess.

[sanmai-NL]

Maintaining multiple Dockerfiles for different contexts is absurd, we have one that generates and runs different bash scripts depending on build target to do the necessary container setup. Other than being absolutely ridiculous that there is no conditional capability in a Dockerfile, it is equally absurd that the only way to specify different .dockerignore files based on target in a docker-compose command is to generate custom files on-the-fly. The assumption that each build target would have a separate Dockerfile is overly presumptuous (there are plenty of completely reasonable configurations that would only need to have a multiple Dockerfile setup solely to address this need for optimizing production targets), and is based on a tooling strategy that is fundamentally flawed (gross reuse of code, a maintainability nightmare). Even if this were not the case, nobody has any business whatsoever forcing this convoluted methodology on the world, and in my opinion it is yet another fail for a technology that was supposed to make it easier to deploy apps across platforms and instead created a whole industry where disillusioned Frankenstein coders have to hack together ridiculously unmaintainable messes to get anything done.

I don't understand the motivation for holding back on adding a -di parameter for docker-compose or even better allowing the .dockerignore path/filename to be specified in the docker-compose.yml file. Why was this not done to start with and why is it even up for discussion?

@Mugane @dtheodor Podman build, though compatible with Docker Engine's build, also supports a Containerfile.in pattern, where reuse is possible.

[jpr5]

This is unbelievable. Can't you just support passing a dockerignore and let users move on with their lives instead of this god awful file name hack? At the moment I am resorting to piping tar output to docker build because the whole configuration of what to include/ignore is a complete mess.

Truly unbelievable. Some "ultimate design" needs to be protected? Utter and complete nonsense. Quality of life is everything -- and every argument that ultimately comes out against making users' lives easier is infuriating and Dumb. And this project is rife with it. These kinds of unforced errors are so disappointing.

8 years this ticket has been around. Yet it's a real, every day problem. Insane. 👎👎👎

[sanmai-NL]

@jpr5 See my previous comment. I moved on to alternatives podman and nerdctl.

[rthomas-sf]

This is unbelievable. Can't you just support passing a dockerignore and let users move on with their lives instead of this god awful file name hack? At the moment I am resorting to piping tar output to docker build because the whole configuration of what to include/ignore is a complete mess.

Absolutely agree with you on that. If the -f flag can be implemented why not the -i? Instead we have this confusing workaround. Now I have to uniquely rename Dockerfiles, that are in completely different directories and make sure the dockerignore files are placed in the context root. Why not give users the flexibility by just giving us the -i/--ignore or whatever.

[Mugane]

Why is this issue closed? Did it get implemented? If not, who needs to be fired?

[bengabp]

Okay, whats the way to implement this, i have a reactjs frontend and a fastapi backend, obviously i dont want my frontend directory to be included in the build stage for my fastapi application and i am using docker-compose

[thaJeztah]

If may depend a bit on your directory structure, but when using BuildKit as builder (which is the default now), docker only sends files that you ask it to COPY or ADD, so for example;

# syntax=docker/dockerfile:1

FROM alpine AS backend
COPY frontend /foo


FROM alpine as frontend
COPY backend /foo

Building either stage will only send either frontend or backend, and will ignore other files.

[bengabp]

my frontend directory is inside the backend directory , so i had Dockerfile.backend and Dockerfile.frontend. but what ive done now is put Dockerfile.frontend inside the frontend directory and call it Dockerfile then create a new .dockerignore file for it , then rename Dockerfile.backend to Dockerfile and then create .dockerignore. so later when building the image , ill have to set the context for frontend image different

[tonistiigi]

@bengabp https://docs.docker.com/build/building/context/#filename-and-location