Multiple Compose files with common anchors do not work

[mitsos1os]

Using the anchor utility of YAML, we cannot use some common defined anchors (such as extension fields, x-*), to declare common configuration in the base compose file and then use them in the extra compose overrides. Docker compose throws error found undefined alias

To reproduce create fragment in docker-compose.yml:

x-function: &json-logging
  driver: "json-file"
  options:
    max-size: "100m"

And use this anchor in a point in a docker-compose.prod.yml:
logging: *json-logging

Running docker-compose -f docker-compose.yml -f docker-compose.prod.yml config will get the error mentioned. Only solution is to duplicate information across all compose files

[shin-]

Yes, anchors are consumed when the JSON file is parsed - this is working as intended.

[alexandernst]

So, anchors are basically not suitable to be used in multiple files? @shin- ?

[shin-]

Yes, they can't be used across different files.

[marioprudhomme]

Is this feature considered in future releases?

[kitzilla]

I was asked to split down my docker-compose.yml to multiples and this lead me to totally abondone extension fields. It would have been nice if the yaml inherit/override anchors from yamls in the left hand side of -f options.

[rulatir]

This is not "working as intended", this is a severe deficiency.

[gautaz]

@rulatir
Anchors are part of the YAML specification which states:

A YAML character stream may contain several documents. Each document is completely independent from the rest.

So this issue (if it is even one) has nothing to do with Docker but with YAML.

[mcabrams]

Are there any recommended ways of achieving the original goal here? I.e. reusing/extending configuration in another docker-compose yaml file? I'm aware that services can be added, but curious about doing things like extending an existing service.

[gautaz]

@mcabrams You might want to call docker-compose with multiple YAML files like this:

docker-compose -f a.yml -f b.yml

b.yml will extend/modify what was declared in b.yml.

This is documented here:
https://docs.docker.com/compose/reference/overview/#specifying-multiple-compose-files

[mcabrams]

@gautaz Thanks -- I'm familiar with and currently use extension by specifying multiple compose files. My question is whether anyone has any strategies for achieving extension via multiple compose files and reusing common configuration of services.

[rbdiang]

@mcabrams Have you found an alternative strategy? I had a similar plan as you, storing all of my common volumes, secrets, and configs as yaml snippets using anchors in a compose.base.yaml file with aliases in my compose-stack.yaml as in:

 docker-compose -f compose.base.yaml -f compose-app.yaml config > app-stack.yaml

I hear what @gautaz is saying, but IMO without cross compose file yaml anchor support the idea of building up a compose file from multiple yaml files isn't very useful 😞.

[Aradiv]

I had the same issue
my solution is instead of using
docker-compose -f docker-compose.yml -f ci/docker-compose.prod.yml build
I'm using
cat docker-compose.yml ci/docker-compose.prod.yml | docker-compose -f - build

This resolves the the aliases inside docker-compose.prod.yml correctly.

There is only 1 small tweek needed

docker-compose.yml

services: &default-services

ci/docker-compose.prod.yml

services:
  << : *default-services

[AdrienPoupa]

This is a limitation of PyYAML, not the YAML format in general.

I think this would work for PyYAML: https://github.com/ChloeTigre/pyyaml-keep-anchors

There also exists an extension for Ruamel.YAML that enables this feature:

https://stackoverflow.com/a/55975390/11115846
https://github.com/Tristan-Sweeney-CambridgeConsultants/ccorp_yaml_include

[Papipo]

@Aradiv your solution works wonders but I can't use it for exec. Do you know how I can?

the input device is not a TTY
ERROR: 1

If I pass -T then I get no output.

[ndepal]

I just ran into this myself. I wanted to load a different logging configuration depending on which additional docker-compose file was specified, where I would have defined those logging configurations as yaml templates.

I solved this instead with extends (wich is supported with compose v3 files as of docker-compose 1.27.0, despite what the documentation says). So now my main docker-compose has a template service that extends another service, the filename of which is an env variable. Depending on how I set that variable, one of two logger configurations is used (loki for production, json-file for development). All of my actual services then use that template service

[Papipo]

@ndepal maybe it's deprecated?

[ndepal]

@Papipo No, more like the other way around. v2 supported extends but v3 did not, as the documentation specifies. After many complaints/requests (see the issue I linked to), it has been added to v3 as well. The note in the documentation that says v3 does not support it (which also links to the same issue) has not yet been removed to reflect the change.

[Papipo]

Oh, that's great

[NobodyIII]

Is this ever going to be fixed or is everyone expected to duplicate information or use the cat file1.yml file2.yml | docker-compose -f - hack? And if we have to use the hack, how do we also use terminal input? Or do we need to use a docker-compose file generator script instead?

As for those who blame it on the specification, this is a severe deficiency that needs some sort of solution, and the only ones currently available are hacks. So figure out a way to accomplish this without violating the specification, or create your own extension to the specification.

[Papipo]

@NobodyIII can't you use extends? It works wonders.

[hirokihokari]

@NobodyIII can't you use extends? It works wonders.

extends was what I was looking for, until the extended service had depends_on.

[computerquip-work]

I know this is old but this is still a pain point. I see no reason this couldn't be worked on. Would patches for this be accepted?

[sammcj]

Yeah it's a shame it hasn't been solved for :(

[typoworx-de]

Typically.... closed without real solution and users asking what's on 🤔
Great ticket management ...

[toby-griffiths]

@typoworx-de I think they have enough to worry about without keeping coming back to discuss things that are out of scope. This is a limitation of YAML and not to do with Docker. I'd rather effort is focused on other useful features, as this can be worked around with some simple file manipulation.

For anyone else coming to this, here are a couple of possible work arounds and an example that works…

1. Write a script to parse the various YAML file, combine then as desired, then dump out the resulting, merged config. If you're in to PHP, there's a great Symfony component for withing with YAML files that would help;
2. Split the shared anchors into a separate file, then concatenate them into temporary files to use with the compose command…

# Note, anchors need to be included first
cat docker-compose.anchors.yml docker-compose.yml > _docker-compose.yml
cat docker-compose.anchors.yml docker-compose.prod.yml > _docker-compose.prod.yml
docker compose -f _docker-compose.yml -f _docker-compose.prod.yml config
rm _docker-compose.yml _docker-compose.prod.yml

… should do it. Works for me after a quick & simple test 😉

[computerquip-work]

@typoworx-de I think they have enough to worry about without keeping coming back to discuss things that are out of scope. This is a limitation of YAML and not to do with Docker. I'd rather effort is focused on other useful features, as this can be worked around with some simple file manipulation.

For anyone else coming to this, here are a couple of possible work arounds and an example that works…

1. Write a script to parse the various YAML file, combine then as desired, then dump out the resulting, merged config.  If you're in to PHP, there's a great Symfony component for withing with YAML files that would help;

2. Split the shared anchors into a separate file, then concatenate them into temporary files to use with the compose command…

# Note, anchors need to be included first
cat docker-compose.anchors.yml docker-compose.yml > _docker-compose.yml
cat docker-compose.anchors.yml docker-compose.prod.yml > _docker-compose.prod.yml
docker compose -f _docker-compose.yml -f _docker-compose.prod.yml config
rm _docker-compose.yml _docker-compose.prod.yml

… should do it. Works for me after a quick & simple test 😉

No offense but this is non-sense. This isn't a limitation of yaml, it's a limitation of docker refusing to use anchors from previously parsed configuration files. This excuse is like saying Docker Compose can't support a new tag because yaml doesn't support it... yaml supports expression of the configuration but what you do with it is completely up to you. It's on Docker to use the information available to it. You ironically gave an example of how this could be done.

When the entire software is based completely on top of yaml, perhaps it would be best to stop using yaml as an excuse to not do something. It makes it looks like yaml is a poor choice for this task when it isn't.

[gautaz]

Hello @computerquip-work,

As already stated in a previous post, you cannot use anchors between YAML documents as each document is independent.

So, from my understanding, @toby-griffiths is right, this is originally a limitation of YAML.

In the meantime, compose added the concept of include which might be what is needed here.

[asdkant-bf]

This is being a really big pain point for me.

I'm trying to have some kind of variable groups, where I can specify what service uses what groups in my base configuration, and then declare the values for the variables in those groups in a single file that overrides my base configuration.

What I managed to do is to have anchors in the override file and use something like this:

environment:
  <<: [*group1, *group2]

But I want to define that association in the base YAML, not in the override. I know I could concievably do this with a bunch of env files, but having the variables spread over multiple files is a no-go.

[luciangabor]

Hoping this helps, I've settled for a different pattern and I think it works pretty well.

The compose.yaml would look like this:

include:
  - path:
      - compose/one.yaml
      - compose/two.yaml
      - compose/common.yaml

In compose/one.yaml and compose/two.yaml I keep everything as simple as possible, but maybe not as simple as this:

---
# one.yaml:
services:
  one:
    image: one
---
# two.yaml:
services:
  two:
    image: two

And then use compose/common.yaml to add various common definitions, and it ends up looking quite ... eloquent:

services:
  one: &idem
    restart: always
  two: *idem

I've also found out that it's best to keep service definitions extendable (without setting depends_on or alike) and "visible" (without setting profiles) and merge such properties in a distinct yaml.
This way I can freely use compose.override.yaml to play around.

In the example above most yamls are inside a subdirectory, but certain yamls, e.g. an env.yaml, could be placed more prominently, near compose.yaml, even if I think this makes it too tempting to edit. :D

$ docker compose config
name: sample
services:
  one:
    image: one
    networks:
      default: null
    restart: always
  two:
    image: two
    networks:
      default: null
    restart: always
networks:
  default:
    name: sample_default