Improve Compose extends and make it less verbose and more powerful.

[Vad1mo]

Currently compose extends is very verbose.

We use extends in the way that we have a base file and then for each environment we replace the significant parts that we want to override or have different.

In the current state of extends if you want to achieve the same you need almost rewrite everything.
Look at out actual example and how verbose it is. There are only two lines that matter.

common.yaml

elasticsearch:
  image: zinvoice/elasticsearch
  hostname: elasticsearch
  restart: always
  dns: 172.17.42.1
  ports:
    - "9200:9200"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro
    - /data/elasticsearch:/opt/elasticsearch/data/elasticsearch

logstash:
  image: zinvoice/logstash
  hostname: logstash
  dns: 172.17.42.1
  restart: always
  ports:
    - "5000:5000"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

kibana:
  image: zinvoice/kibana
  hostname: kibana
  dns: 172.17.42.1
  restart: always
  ports:
    - "5601:5601"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

logspout:
  image: zinvoice/logspout
  hostname: logspout
  command: logstash://logstash.docker:5000
  restart: always
  dns: 172.17.42.1
  ports:
    - "8003:8000"
  volumes:
    - /var/run/docker.sock:/tmp/docker.sock

doorman:
  image: zinvoice/doorman
  hostname: doorman
  restart:  always
  dns: 172.17.42.1
  ports:
    - "8085:8085"

child.yaml

elasticsearch:
  extends:
    file: ../common.yml
    service: elasticsearch

logstash:
  extends:
    file: ../common.yml
    service: logstash

kibana:
  extends:
    file: ../common.yml
    service: kibana

logspout:
  extends:
    file: ../common.yml
    service: logspout

doorman:
  environment:
    - DOORMAN_GITHUB_APPID=xxxxxxxx
    - DOORMAN_GITHUB_APPSECRET=xxxxxx
  links:
    - nginxtrusted
  extends:
    file: ../common.yml
    service: doorman

With the new extends feature one would add a global_extends/import like this:

extended_child.yaml

global_extends: 
 - common.yaml
 - extra.yaml 

doorman:
  environment:
    - DOORMAN_GITHUB_APPID=xxxxxxxx
    - DOORMAN_GITHUB_APPSECRET=xxxxxx

Thats it.

Advantages

1. A less verbose code is easy to maintain and understand
2. You can add remove service in the base.yaml without adding/removing entries in the children. (Missing a container in Prod because someone forgot to ad it as into the prod.yaml.
3. You can aggregate services into one.
4. The configuration can be easily created by CI or or maintained by an application.
5. You have a nice overview what gets set and you can check in this in a repo.
6. This will make Interpolate environment variables in docker-compose.yml #1377 and Pass variables inside fig.yml on runtime #495 obsolete and 75 +1er happy :)

Main featues

• append nodes that are declared in parent yaml
• modify/overwrite nodes declared in parent
• add new nodes
• the only thing you can't do is to removed nodes
  see examples in Improve Compose extends and make it less verbose and more powerful. #1380 (comment)

[dnephin]

I think this is very similar to what I proposed in #318 , and it's a feature I'm still very interested in. There's a bunch of complexity with something like this, so it's taking a while to get there, but I think we're getting closer.

[aanm]

Also #1361, but I agree, the optimal would be docker-compose up --extends ../common.yml and that's it, all the merge would be automagical since the services' names would be the same in ../common.yml and docker-compose.yml

[Vad1mo]

indeed #318 and #1361 has the same intention. @dnephin I am wondering why it is such a complex task? Does this go beyond merging two yaml files into one tree.

[aanm]

@Vad1mo Don't forget that extends merge both configuration depending on each option. For example:

In the case of build and image, using one in the local service causes Compose to discard the other, if it was defined in the original service.

(@Vad1mo you have a malformed link on the previous comment)

[andrerom]

"global_extends" could just be called "import", here is format used in Symfony framework, addopted/simplified a bit:

# Any config blocks imported here can be selectively be changed below
# wouldn't normally have one file per service/container, this is simplified example
imports:
  - ../parameters.yml
  - app/rails.yml
  - db/mysql.yml

# override some stuff
rails:
  environment:
    SESSION_SECRET: SomeSecret

parameters.yml

# Additional concept: parameters, don't need to change on each machine where the app is deployed
# http://symfony.com/doc/current/best_practices/configuration.html#application-related-configuration
# Would typically be the place to use env variables and defaults when #1377 is in, to keep it in 1 place
parameters:
  app_env: prod

rails.yml

rails:
# (...)
  links:
    - db
  environment:
    RACK_ENV: "%app_env%"

mysql.yml

db:
# (...)

[Vad1mo]

@andrerom
There is a conceptual difference between extends and import. Import is for composing and orchestrating services together to create something new/big out of small peaces.
Example: Frontend + backend + db merged together into one service file to start everything together.

On the other side extends means you want to extend or modify/override the behavior of the services you already have in place. You extend something for a specific situation.
Example: add/change port or set env var. of prod.

[andrerom]

I agree, but then what you imply is that your global_extends does not really import any of the services, so your example would need to be extended so you define all services again (with a extends.service part, just saving repeating the extends.file part). So your example to me looks more like an import, where you redefine part of the doorman service, exactly like imports does in my example.

[hoIIer]

@Vad1mo @andrerom I am new to docker/compose etc and working on a new microservice based project... I'm running into issue where I wanted to have a top level repo with just a docker-compose.yml that orchestrates all the pieces together e.g.

Example: Frontend + web gateway service + content service + user auth service + async service etc together into one service file to start everything together.

It seems that the current extends implementation doesnt support links from child compose files so I'm wondering if what you guys are talking about applies to this case (a master compose.yml that ties all the services together and individual compose.yml files within each service that is decoupled mostly and can have links etc)

Just curious thx!

[Vad1mo]

@erichonkanen what you describe was initially not in my intention but the proposal would also allow this kind of behavior.

[aanand]

@erichonkanen What you describe is something we've talked about before - see #758 - and which we should continue to explore.

[aanand]

I'd like to think more about "global extends". It would be a much less verbose solution to the problem of configuring an app for multiple environments, which is one of the primary things users are asking for better ways to do.

Let's imagine the simplest possible implementation: you can import all services from a single Compose file, and can then add new services and reconfigure imported ones.

base.yml

web:
  build: .
  links:
    - db

db:
  image: postgres

docker-compose.yml

EXTENDS: base.yml

web:
  ports:
    - "8000:8000"
  volumes:
    - .:/code

production.yml

EXTENDS: base.yml

frontend:
  image: registry.mycompany.com/reverse-proxy
  ports:
    - "80:8000"
  links:
    - web

web:
  image: registry.mycompany.com/webapp

I'm a bit scared of this feature because of the amount of implicit-ness, but it seems to be the only sensible way to avoid the huge amount of boilerplate currently needed to make small changes to a large file.

As for extending multiple files (as in @Vad1mo's example), the potential for confusion there seems much greater. What use cases would it serve?

[hoIIer]

@aanand I am wondering if the use case of a microservice based architecture is a use case? Currently Im working on a new project using both docker and microservices for the first time... compose is a neat and convenient tool but the one thing I wish it had were the ability to decouple extended services/repos more cleanly, aka not have to put the links in the top level docker-compose.yml.

Something like this would be sweet:

orchestrate.yml

service-content:
  import:
    file: ../service-content/docker-compose.yml

service-auth:
  import:
    file: ../service-auth/docker-compose.yml

service-accounting:
  import:
    file: ../service-accounting/docker-compose.yml

service-content/docker-compose.yml

web-db:
  image: postgres:latest

web:
  build: .
  command: python manage.py runserver 0.0.0.0:8000
  volumes:
    - .:/code
  ports:
    - "8000:8000"
  links:
    - web-db

Which would make containers prefixed by the top-level import name e.g. service-content_web-db_1

What are your thoughts on something like this? Right now we are definining everything in the orchestrate.yml file which works but it couples everything to that file/repo whereas it'd be ideal to allow each service to define it's own config/build/link instructions etc...

[andrerom]

@aanand I think this would work.

In my use case, what I need is to be able to setup a app cable of using several different services depending on config.

examples: using postgres instead of mysql in one specific setup and configure app container for this with env variable + optionally adding memcache or redis as caching and inject app config + optionally adding solr and inject app config + being able to do what you showed above, adding varnish/reverse-proxy, again injecting config to app for that so it can purge cache.

This is why I was thinking import as it essentially allows the misc services to be defined separately, and the misc combinations import and configure as they seem fit. But I think it can work with what you propose also, with base being small and the more complex setups gradually extending it layer by layer.

[aanand]

@erichonkanen That's an interesting idea - creating multiple tiers of nesting - but based on your example I think we're now talking about a different feature entirely. It sounds a bit more like what @dnephin proposed in #318, where multiple independently-runnable apps are brought together.

[hoIIer]

@aanand ok yeah that makes sense, I was thinking about it more and it seems slightly different

[aanand]

@andrerom your suggestion sounds a lot like Compose's existing extends option. What are the primary differences, as you see it?

[aanand]

Important note: the global EXTENDS option lets you extend files with links in, as opposed to the single-service extends option, which doesn't let you extend services that link to others.

(The same applies to the other types of inter-service dependency, volumes_from and net: container)

[Vad1mo]

@aanand regarding your #1380 (comment)

as in my first question multiple extends would allow to combine services allowing to deploy a smaller set of services on the developers machine for example or you could deploy only subset of services to a machine. Multiple extends is IMHO not a top priority for us.
Top Priority for us is to reduce the current verbose boilerplate code which is too error prone. Currently each litte change need a modification in multiple files.

[scottbelden]

@aanand Would your proposal allow for services that extend a base file to have a different name? For example, I would like to have a basic "connection" setup and all my services could use that connection.

conn.yml

conn:
  links:
    - db
    - mq

db:
  image: postgres

mq:
  image: rabbitmq

serviceA.yml

EXTENDS: conn.yml

serviceA:
  # somehow specify to use the conn links
  image: imageA

serviceB.yml

EXTENDS: conn.yml

serviceB:
  # somehow specify to use the conn links
  image: imageB

[Vad1mo]

@scottbelden in the child compose file you could basically

• extend nodes declared in parent
• modify or overwrite nodes declared in parent
• add new nodes
• the only thing you can't do is to removed nodes

[andrerom]

@andrerom your suggestion sounds a lot like Compose's existing extends option. What are the primary differences, as you see it?

@aanand you are right, combination of service and global extends will allow it without to much boilerplate. I was to stuck in the import semantics.

[dnephin]

@aanand I see this feature as basically the same idea as #318. The terminology is a bit different, but the goals and implementation are basically identical.

EXTENDS would just be a more restrictive version of "include" or "import". EXTENDS forces you to have a chain of files, where as import would let you pull in unrelated things into a single composition (instead of trying to force them into some hierarchy). It's basically inheritance vs composition, and I always opt for composition.

[hoIIer]

maybe this is superficial but why the all caps EXTENDS? any alternatives? also I like the idea of composition via include/import

[Vad1mo]

@dnephin full ack for your #1380 (comment) the goal is the same the approach is different. The only advantage there might in favor of EXTENDS might be the ability to cope better with override conflicts and cycles.

Bu to be honest in the meantime I don't care what is "the" best solution. Anything (#318, #1380 or #1377) better then the status quo would already improve the situation by a magnitude for many here.

Its ok for me to close this proposal and move to #318

[aanand]

@dnephin I agree there's a lot of overlap (and that composition usually beats inheritance), but how would #318 serve the use case of wanting to make small changes to an app's configuration to tailor it to a particular environment? Do you have any ideas for syntax/semantics there?

[Vad1mo]

finally, I see it that main difference between this proposal and #318.

• append/extend content in nodes that are declared in parent yaml
• modify/overwrite nodes declared in parent

Here are the two examples that of the above features:

parent.yaml

doorman:
  image: zinvoice/doorman
  hostname: doorman
  dns: 172.17.42.1
  ports:
    - "8085:8085"

append.yaml

global_extends: 
 -  parent.yaml

doorman:
  restart: always

overwrite.yaml

global_extends: 
 -  parent.yaml

doorman:
  hostname: doorman_prod

@dnephin can this be achieved with #318 ?

[dnephin]

We already have a way to extend services, so if we were to add a way to include other compositions, it seems like it would be possible to continue using the existing mechanism for extension of services.

[Vad1mo]

but the current way is quite verbose. As you can see from my initial proposal you basically double your code base just to make some modifications. I am not sure if the current extends could be modified in a way to reduce the verbosity and adding the appen/modify functionality.

[aanm]

@Vad1mo Why isn't the @aanand proposal of having a "GLOBAL" keyword good enough?

[Vad1mo]

@aanm you mean from his #1380 (comment) ?

I see @aanand proposal as a variation/explanation to my initial proposal. The only difference is the limitation to extend from only one parent file.

[dnephin]

@Vad1mo includes are just a different model. You wouldn't have the same configuration. common.yaml would actually have links between things, you wouldn't need extended_child.yaml at all. All you'd have to do would be create a new doorman service and have it link to the appropriate services in common.yaml

doorman.yaml

include: common.yaml

doorman:
  image: zinvoice/doorman
  hostname: doorman
  restart:  always
  dns: 172.17.42.1
  ports:
    - "8085:8085"
  environment:
    - DOORMAN_GITHUB_APPID=xxxxxxxx
    - DOORMAN_GITHUB_APPSECRET=xxxxxx
  links:
    - nginxtrusted

If you need some other variation of dooman, you'd create doorman-prod.yaml and include common.yaml again. This is where the comparison between composition vs inheritance comes in. You're no longer inheriting things, you're breaking them up into components and sharing them that way.

[aanm]

@Vad1mo yes

[Vad1mo]

@dnephin thanks for the explanation. As I see #318 now, it will not reduce the current verbosity or contribute to DRYness when you want to orchestrate services for different environments or settings.

It also won't be an elegant alternative to #1377 as I had in mind with this proposal by allowing to have only one concept of stacking/extending pretty much like docker layers.

[bfirsh]

@Vad1mo Thanks for the feature suggestion! We should make this happen.

I've tried to aggregate the feedback from this thread into #1987. Let me know if you think I've missed anything.