Docker compose support

[deviantony]

Give the user the ability to start a docker-compose stack by uploading their docker-compose.yml.

Specs update (22/04/2017):

Ability to define stacks inside Portainer with support to docker-compose.yml files (v2 & v3).

New UI views: stacks (A), stack-details (B), stack-creation (C)

A. stacks view

• List existing stacks within a table (stack name)
• Access to stack-details by clicking on the stack name
• Ability to delete one or more stack(s) (will also remove the associated containers)
• Ability to execute actions against a stack (down, up)

B. stack-details view

• Give details about the stack (name, status)
• Ability to delete the stack (will also remove the associated containers)
• Ability to execute actions against the stack (down, up)
• Ability to execute actions against the stack's services (create, start, stop...)

C. stack-creation view

• Ability to create a stack (fields: name, compose file -upload or URL to file)

[jslatten]

This would be awesome and make Portainer a real competitor with Rancher if you can integrate into the template profiles.

Any thoughts on creating a portainer container to package swarm as an easily deployed "agent"?

If there is interest I can contribute...

[deviantony]

@jslatten can you describe what do you mean by 'integrate into the templates profiles' ?

Also, I don't think there's a need to 'package' Swarm at the moment, especially with the arrival of Docker swarm mode, it has never been easier to deploy a Swarm cluster.

[jslatten]

I meant that the templates could reference docker-compose templates instead of basic references to standard docker images

[deviantony]

Yes, the way I see it it would be to add another section in the app 'stacks', 'bundle' or something where you would be able to upload your compose file to start the associated stack.

[scher200]

great idea

[ysilvela]

Use compose files and deploy multiple images linked its the most useful in dockers enviroments

[slattery]

@deviantony thanks for setting this on a milestone.

[paulkling]

I think it would be interesting for Portainer to do a GET from a URI for the compose file, file upload, or straight up typing one in.

[scher200]

Then to have a vew options like: run, up, down, rm and etc.. 2016-12-14 22:17 GMT+01:00 Paul Kling <notifications@github.com>:

…

I think it would be interesting for Portainer to do a GET from a URI for the compose file, file upload, or straight up typing one in. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#257 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJG6XY0arR1jyBwB_ojQrgWqwwogluhKks5rIFztgaJpZM4KOcDQ> .

--

[lavvy]

Yeah. get_from_url option should please be included

…

On Dec 14, 2016 5:46 PM, "scher200" ***@***.***> wrote: Then to have a vew options like: run, up, down, rm and etc.. 2016-12-14 22:17 GMT+01:00 Paul Kling ***@***.***>: > I think it would be interesting for Portainer to do a GET from a URI for > the compose file, file upload, or straight up typing one in. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#257# issuecomment-267159246>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AJG6XY0arR1jyBwB_ ojQrgWqwwogluhKks5rIFztgaJpZM4KOcDQ> > . > -- — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#257 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGsC46DElksfsM6dbScPmxlJ9-gHWzC0ks5rIHGzgaJpZM4KOcDQ> .

[mazerty]

as an active user of compose I'd really like such a feature 👍

[thomas15v]

I wouldn't mind getting rid of the current template system and just move to compose 😄. It's in essence exactly the same, but with more options.

[scher200]

yeah it would work wonderfull with: composerregistry

Search API
full text search: https://www.composeregistry.com/api/v1/search?query={QUERY}

implementation for search would be equal to:

e.g. curl -H 'x-key: default' https://www.composeregistry.com/api/v1/search?query=elk

{
"items": [
  {
    "name": "docker-elk",
    "owner": "deviantony",
    "stars": 345,
    "id": "AVBCF9PggLKoASuOwp_8"
  },
  {
    "name": "st2contrib",
    "owner": "StackStorm",
    "stars": 45,
    "id": "AVBR45mWwBuN0C9fxHc5"
  },
  ...
  ]
}

[schmunk42]

Compose support would be truly awesome.

I wanted to give some examples how we use compose on a swarm in production, see screenshot below:

(env = swarm environment)
(rectangle = docker-compose-stack)
(arrow = merging, default config, external files)

Actually we're only using native features, such as compose .env-files, container-env-files or merging multiple files. It would be nice if we could organize files in different folders.

Btw: we've created the swarm with roj, which is fully compatible to portainer, since both use native Docker tools.

[deviantony]

How would you guys like to have compose support in Portainer? The way I see it at the moment :

We should give the users the ability to define 'stacks'. When creating a stack, you provide a name and a docker-compose YML file. Once the stack is created Portainer should give you the ability to manage it (up down etc) and also a way to manage each service in that stack (start stop a service etc). From the stacks view you should be able to manage your stacks (add, edit, delete) and access to a detailed view for each stack (which allows you to manage it as well as giving a visual representation of the containers of the stack).

[ncresswell]

Stack, or Apps ? I thought that "stack" as a term, is being used in Docker 1.13. Rgds, Neil Cresswell On 15/01/2017, at 10:04 AM, Anthony Lapenna <notifications@github.com<mailto:notifications@github.com>> wrote: How would you guys like to have compose support in Portainer? The way I see it at the moment : We should give the users the ability to define 'stacks'. When creating a stack, you provide a name and a docker-compose YML file. Once the stack is created Portainer should give you the ability to manage it (up down etc) and also a way to manage each service in that stack (start stop a service etc). From the stacks view you should be able to manage your stacks (add, edit, delete) and access to a detailed view for each stack (which allows you to manage it as well as giving a visual representation of the containers of the stack). - You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_portainer_portainer_issues_257-23issuecomment-2D272659965&d=DQMFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=Fw9ndRCBfdaqM_QJPyMAN8dnAJGj7OhptUL9ydxCALg&s=p25fvB-hAknn38kSn_OEJqEqB3TuGZ8-UqFZwealFtI&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AWGrlTwoVUv-2D5p-5FIce0kU9jEY68YRJF0ks5rSVR1gaJpZM4KOcDQ&d=DQMFaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=Fw9ndRCBfdaqM_QJPyMAN8dnAJGj7OhptUL9ydxCALg&s=V2ivYy3sHro5tBndkdegDkzw5Q8cGUU0pQXiAJjAFWc&e=>.

[deviantony]

I used the term 'stack' with some quotes as an example. We might need to figure out another term.

[dimkk]

@deviantony I am using https://github.com/francescou/docker-compose-ui with portainer.

I like the way it behaves, just like you've described.
Don't got any spare time now, but would have tried to embed features of compose-ui to portainer.

[schmunk42]

How would you guys like to have compose support in Portainer? The way I see it at the moment :
We should give the users the ability to define 'stacks'. When creating a stack, you provide a name and a docker-compose YML file.

From my point of view I'd suggest rather to point to a directory containing the corresponding files. A docker-compose configuration can consist of several files, not only one docker-compose.yml, but i.e. a .env defining the .yml files to include and a project name.
Moreover it would be very important to connect containers to their docker-compose "stacks" - that can be done by parsing the com.compose labels of the container.

I am using https://github.com/francescou/docker-compose-ui with portainer.

I know that project and it is nice, definitely worth a look.

Stack, or Apps ? I thought that "stack" as a term, is being used in Docker 1.13.

The problem with Docker sometimes is that they mix their own terms, while a stack is now much more related to Docker-cloud (see also stackfiles) the platform started out as place for sharing docker-compose.yml files.

Or this one called neam/docker-stack - stared on Mar 1, 2015.

But yes, we should differentiate it clearly :)

[deviantony]

Moreover it would be very important to connect containers to their docker-compose "stacks" - that can be done by parsing the com.compose labels of the container.

Could you give us more details about that one? Would you like to be able to see the associated 'stack' in the container view if a container is part of a compose 'stack' ?

[schmunk42]

Even both ways, so from stack -> container and from container -> stack

For example, if you inspect a container, you'll get

"Labels": {
    "com.docker.compose.config-hash": "0ac7cfd1b1be53c964dfcc90c728dbc543493e5d38cc34822676b274822914b5",
[...]
    "com.docker.compose.project": "myproject",
[...]
}

I would be very helpful, if containers are grouped by their project-names (myproject), even if you have no information about the "stack" (project) itself.
If you would create/upload a stack-folder containing a docker-compose.yml and .env file, you could kind of attach the config to the running containers.

Moreover, Portainer could warn you, if you're using a COMPOSE_PROJECT_NAME twice, or if there are illegal characters set, i.e. there are no - or _ used for the container name.

[scher200]

yes, this would ensure a good view on your 'stack', no matter how you ran it.

[lobre]

Really looking forward to have compose integrated into portainer as well!

An interesting feature in addition to compose would be the notion of questions as Rancher does.

See: https://docs.rancher.com/rancher/v1.2/zh/catalog/private-catalog/#questions-in-the-rancher-composeyml

It is really handy to populate the compose file when it has to contain secret data or specific configuration.

[allamand]

I would prefere the way followed by docker-compose-ui that would be to set the secrets data in a .env file read at same time of docker-compose.yml. As in compose-ui, it would be great to specify a git url which could contains directories for each stacks instead of loading one by one and then the git still be the real source of the compose file

[lobre]

The .env file is not specific to docker-compose-ui but it is native from compose (https://docs.docker.com/compose/environment-variables/#/the-env-file). So maybe the idea would be to implement an UI to fill in this .env file. That would look like to this "question" feature integrated in rancher-compose.

This is not mandatory but that would be a nice feature ;)

And just a few questions that come into my mind when thinking about a implementation of compose in portainer:

• Compose would need to be installed into portainer container?
• What would be the library used to interact with compose (as it does not provide any standard REST api...)? compose-ui uses directly python calls wheareas Rancher uses libcompose (that may be more appropriate as written in go).

[deviantony]

@loric-

About your questions:

Compose would need to be installed into portainer container?

No, Portainer will take care of interfacing Docker Compose

What would be the library used to interact with compose (as it does not provide any standard REST api...)? compose-ui uses directly python calls wheareas Rancher uses libcompose (that may be more appropriate as written in go).

Yes, we will probably use libcompose. I already had a look at it and it seems easy to integrate inside Portainer.

[deviantony]

@samprog1 docker-compose and docker stack deploy are two different features and both cannot be managed via the Docker API. We still need to check if we can integrate docker-compose the same way we did for docker stack deploy.

[mister2d]

@deviantony stacking works great for my swarm cluster. Thanks for explaining.

There is one thing I'm still struggling to understand with the compose support. I use a private registry (with auth) that is defined within Portainer. How do I have my deployed stack pull from that registry? I get errors every time I deploy with private registry entries for my images.

Thanks.

[deviantony]

@mister2d this is not available in 1.15.0, we implemented it recently (see #1313) and will be available in 1.15.1 (to be released soon).

In the meantime, you can try it in portainer/portainer:develop (linux amd64 build).

[Toilal]

Does it support building image ?

[deviantony]

Well, that should be something we should have a look at, you mean the equivalent of docker-compose build ?

[Toilal]

Yes, supporting such docker-compose.yml files

version: "2"
services:
  web:
    build: ./docker/web
    ...

This should build image located in ./docker/web relative to the docker client working directory.

Problem is that Docker API doesn't support sending the build context over API. It only supports downloading it from a tarball URI, which can make things quite complicated.

See https://docs.docker.com/engine/api/v1.33/#operation/ImageBuild

remote: string
A Git repository URI or HTTP/HTTPS context URI. If the URI points to a single text file, the file’s contents are placed into a file called Dockerfile and the image is built from that file. If the URI points to a tarball, the file is downloaded by the daemon and the contents therein used as the context for the build. If the URI points to a tarball and the dockerfile parameter is also specified, there must be a file with the corresponding path inside the tarball.

[bruno-lopes]

Any news on this?

[samprog1]

Docker Stack support was implemented. Your Docker host needs to be in swarm mode for this feature though.

Docker compose was not yet implemented

[gjonespf]

I think what we have here is already awesome. I have a similar use case to some above using a set of folders per stack (e.g. @alwaysastudent, @schmunk42 ). I'd suggest to get something like that to work, would be great to build a custom "stack manager" container that interfaces to portainer using the API (API PR and potentially separate project for the manager), rather than adding this complexity to the core project. I'd imagine you could build something very basic which links into app templates and just uses a list of folders/compose files which you could volume mount / build into the "stack manager". I can also see for my use case I'd probably want something more custom as my current stacks use powershell deployment scripts. I'll have to have a play around with the API and see what is available.

[durino13]

Any news on docker-compose support? It would be nice to have the ability to start / stop / restart the whole compose stack ..

[Doctor-Who]

I hope it will be available with load balancing option

[iotgopher]

Any news on docker-compose support?

[ncresswell]

We will not be adding compose support, instead we will focus on extending stack support. Rgds, Neil Cresswell On 11/02/2018, at 12:51 PM, rtmaster <notifications@github.com<mailto:notifications@github.com>> wrote: Any news on docker-compose support? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_portainer_portainer_issues_257-23issuecomment-2D364724146&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=fjctoKmoiDjEKvN0vAjtGyk-e7veH02nkv_oWuJyo9g&s=oUMEJlbUVvsU329BdxgNIXwx4HkXALFwBPLrPVmuIKU&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AWGrlQV-2DO-2DCftdiSig2ma4E4-5FaU4l3mxks5tTn-5FsgaJpZM4KOcDQ&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=0fx0h4vB56iTLpw2McH1ZD6TqG_QGpbggVOB-PfMJpM&m=fjctoKmoiDjEKvN0vAjtGyk-e7veH02nkv_oWuJyo9g&s=EWWxopT1zWm5twM4X9Jcz4L724wqzgH2N2mJuj1XJxI&e=>.

[mrmachine]

This decision makes sense. A regular compose file started on a standalone Docker Engine has no corresponding entity that is listable or manageable via Docker CLI/API. If there were, it would be a functional subset of a Swarm Mode stack (lacking the deploy options).

It would be very confusing if Portainer's UI tried to implement Swarm Mode stacks (docker stack ...) AND a faux "compose stack".

Of course, this means everyone who runs on a single host (vs a cluster) will probably want to run a single node Swarm Mode cluster just so they can manage stacks. Doesn't seem like a big deal.

This issue should be closed, then?

[ronlut]

What about multiple compose files, like @schmunk42 showed? Is that somehow supported?
We are using portainer and started using multiple compose files using docker stack deploy -c docker-compose.yml -c docker.compose.prod.yml stack_name etc.

[deviantony]

@ronlut this is not supported yet but you can open a feature request for this.

[HarvsG]

Any update on this?

[silvether]

Wanted to post a quick workaround to this:

1. docker-compose up
2. Open Portainer web-ui and go to the container
3. Select Duplicate/Edit
4. Remove the stack from the LABELS section where is says com.docker.compose.project
5. Deploy edited container

-- done