Add the ability to update and redeploy a stack created from a git repository

[dcharbonnier]

It would be useful to keep the git url and the credential of a stack deployed with a git url and to provide a api/ui to trigger an update.
Configuration of a stack is as important as code and should be persistent and versioned, git is the perfect tool for that.

[deviantony]

Do you mean the ability to trigger a git pull and then redeploy the stack?

[dcharbonnier]

Yes

[rahulruikar]

@deviantony I am thinking of

• Slider "Authentication"
  - Username
  - Password
• Rename existing button label from "Update the stack" ==> "Update"
• New button with label "Redeploy"
  what do you think ?

[deviantony]

In my opinion, the credentials should be persisted in the stack data.

If the stack was created from a repository, a new button should be added "Pull and deploy".

[rahulruikar]

Update..
@deviantony I will go ahead with approach mentioned in your comment.

@deviantony I didn’t go with this approach because credentials could change Between saving with data and redeploy attempt(rare cases but possible). Also in one of thread you mentioned that portainer does not save these github credentials.

[vladbabii]

I would also like this feature. Until then I wrote a small script to update stacks and hooked it into git - https://github.com/docker-how-to/portainer-bash-scripts

[OmgImAlexis]

@rahulruikar if you're adding basic auth it'd be great to also be able to use a (read only) SSH key like how Github has deploy keys.

[deviantony]

Following the discussion in #1793 (comment)

As I'm not keen to store unencrypted credentials for git repositories inside Portainer I think that the way to go is to implement #1752 first and then work on this one.

[rahulruikar]

Ok..I will start working on #1752 this week.

On Thu, 26 Apr 2018 at 12:18 am, Anthony Lapenna ***@***.***> wrote: Following the discussion in #1793 (comment) <#1793 (comment)> As I'm not keen to store unencrypted credentials for git repositories inside Portainer I think that the way to go is to implement #1752 <#1752> first and then work on this one. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1753 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHXNPSgOjCLCHiH5wMz2reZV7Jg52g8hks5tsIWmgaJpZM4S3Cmi> .

-- Sent from Gmail Mobile

[shyd]

Hi,
I am looking forward to have this option as it would optimize my deployment process! 👍
Two things I have in mind for this:

1. Will sidecar files work? There are some config files I need to mount into the container for e.g. nginx.
2. I'd love to have the feature to trigger these redeployments thru webhooks from dockerhub or CI/CD.

Cheers

[deviantony]

@shyd

Not sure about the sidecar files, but if they're located inside the repository and part of the stack deployment should work.

And yes, webhooks for this feature is in our plans.

[somq]

Hi @deviantony
Any news on this?
Is it on your road map yet? If yes, do you eventually have any eta?

[stevensbkang]

@somq I am currently working on this feature - will let you know once it's ready for testing :)

[somq]

@ssbkang great news thanks!
Will be happy to test it, if you need any more help let me know. I can test it as soon as API endpoints are reachable just to let you know.
Just to understand, do you have any idea if it's a matter of days, weeks, more?

[upcFrost]

will it be one-way (re-deploy) or two-way? I mean, if you change something in the UI, will it commit/push to the repo as well, or it will only be able to pull and re-deploy but not push?

I agree, pushing a commit should save a lot of work as it simplify the project management.

To be fair, with one-way sync I don't really see the point of linking portainer to git. It will make migration to portainer easier, that's true, but it basically means losing the changes history with this migration. Or just using portainer as a read-only UI, which would be a bit strange

[Mr-Markus]

This Issue was opened 3 years ago and is very popular. But until now this feature was not implemented.
I read the discussions and @deviantony said it will be there in Q2 of 2021.

Is there a beta version of it at the moment or where can I find that part of code to test it?

[deviantony]

@Mr-Markus there is no beta available at the moment, the development on this piece is planned to start next week! We're targeting CE 2.5 at the end of May but as we have many things in our work pipe and a few delays have happened already it might be part of CE 2.6 at the latest.

We're considering it as one of the most important feature to be added next so it will definitely happen soon!

[chrisbecke]

I was just about to open a new ticket when I deployed a stack using git Repository, and then it didn't offer a re-deploy after I'd made changes option. And I'd made deploy tokens and everything.

[fdelucchijr]

For me, it would also be a very useful feature.
Also nice to have a webhook for updates of the stack created from git.

Same here! It would make a lot more easier to deploy and mantain the stacks and configurations of the app.
Note: Also for deployments with a more complex .env file support for GitHub Secrets will be awesome!!!!

[HazeTheMaze]

@deviantony It didn't make CE 2.5? When will CE 2.6 be released?

[lukasmrtvy]

@HazeTheMaze Do yourself a favour and use some proper CI tool ( like concourse ) and https://github.com/docker/compose-cli. Then You will get UI, gitops, autosync, credentials, notifications...

[deviantony]

@HazeTheMaze this did not make it to CE 2.5.0 and we're now targetting CE 2.6.0, I will share a preview version of this feature when this is available as we are still working on this.

[deviantony]

For those following this thread, we now have a preview version available via the following image: portainerci/portainer:pr5139

Happy to hear your feedback about it if you have some time to give it a try.

This is a development build, it is not recommended to run it in production environment.

[MooRogue]

Tried it out! Good start! Deployment and updating functionality worked just fine here.

Minor comments:

• Username/access token is not saved so... updating from a private repo isn't too smooth
• Seems to a minor bug where if you try to update from a private repo without providing a user/password it loses track of the docker-compose file on the local filesystem (or maybe erases it? Need to experiment)
• As previously mentioned, the ability to push changes up would also be awesome 😀
• Wondering if a future enhancement to periodically check for changes to the git repo and automatically redeploy would be considered... or would that be stepping too much into the CI/CD realm

[gabschne]

Wondering if a future enhancement to periodically check for changes to the git repo and automatically redeploy would be considered... or would that be stepping too much into the CI/CD realm

Would be awesome. We have a repository with all the Compose files we need for the company's internal (IT) services. Of course you can do that with CI/CD, but would definitely be easier if Portainer could do it. And if periodic checks should be implemented, one additional feature request: It would be perfect if you could schedule the check (e.g. every saturday at 3am after the nightly backup and when the employees are rather not working).

Anyway, the ability to redeploy from a git repo is already a big improvement, so thank you!

[kumy]

Aside from a "polling" way to detect a change in the repo, webhook called from github/gitlab/whatever on commit/PR would be nice too.

[waja]

I think using a webhook would be a more common way. It provides the possibility to deploy changes instantly when changes occurs.

[tfenster]

@deviantony certainly looks great! Some questions/feedback:

• "prune services" should imho not be hidden in the "Editor" tab. I might want to use that even if I am not editing the stack definition at all
• It would be great if we had an option to save the auth for a private github repo. The scenario for us would be that person X with access to the private repo sets up the stack and then person Y without the need to have access to the repo could update the stack. Is that something you want to support?
• Setting env variables seems to work only on the initial deployment. If I redeploy, I can still see them in the editor, but the deployment doesn't apply them and they are gone in the editor afterwards as well

[fdelucchijr]

Deployment its perfect, the only annoying thing is Git Auth introduction in every pull, adding support of storage a global git credentials or per stack would be perfect! 👌🏻
Apart from that, thank you!!! this feature is something I have been waiting even before knew I needed it.

[deviantony]

Thanks a lot for all the feedback everyone!

We're aware that this version is not perfect but we think it is a good base to start with and we'll continue enhancing it.

@MooRogue

Username/access token is not saved so... updating from a private repo isn't too smooth

I agree, I'm not a huge fan of the UX but it's a very simple base that we can build on in the future (I would like us to introduce a proper Git provider integration in the future allowing you to authenticate to a 3rd party git provider - github, gitlab...- and persisting only the required information).

In the meantime, we're starting with this and we'll introduce the ability to persist it with a future enhancement.

Seems to a minor bug where if you try to update from a private repo without providing a user/password it loses track of the docker-compose file on the local filesystem (or maybe erases it? Need to experiment)

We'll have a look to see if we can confirm that one, again this is a development build so there might be a few other bugs here.

As previously mentioned, the ability to push changes up would also be awesome

This is not something that we have in our plans, at the moment, our current vision is that if you have a stack definition stored in Git, you should always update it in Git (supporting pushing might introduce sync issues). But I'll keep it track of it.

Wondering if a future enhancement to periodically check for changes to the git repo and automatically redeploy would be considered... or would that be stepping too much into the CI/CD realm

We already have a piece of work in progress that is a continuation of this feature and that will allow you to toggle automatic update of the stack, by toggling this switch on you'll be able to:

• Periodically pull and update
• Use a stack webhook to be set in your CI/CD - cc @kumy and @waja

Here's a quick glance at our design for it:

@gabschne this might be of interest for you as well, I'll keep your feedback aside for an enhancement allowing a more advanced schedule.

This enhancement would also allow you to persist git credentials (we recommend the use of a personal access token here) - cc @fdelucchijr

@tfenster

"prune services" should imho not be hidden in the "Editor" tab. I might want to use that even if I am not editing the stack definition at all

Thanks, indeed it might be interesting to have prune action now instead of our previous "Update and optionally prune". I'll keep track of this enhancement.

It would be great if we had an option to save the auth for a private github repo. The scenario for us would be that person X with access to the private repo sets up the stack and then person Y without the need to have access to the repo could update the stack. Is that something you want to support?

Yup, see my comment above regarding our following enhancement of the feature.

Setting env variables seems to work only on the initial deployment. If I redeploy, I can still see them in the editor, but the deployment doesn't apply them and they are gone in the editor afterwards as well

Thanks, we'll investigate that one.

Thanks again for the feedback everyone!

[GlassedSilver]

Well... now relative paths are an absolute necessity, aren't they? Talking about #2046

The horror to imagine that doing all the manual leg work to deploy a stack - necessary for every update...

Completely throws out UX, automation and TBH the error was a bit of a challenge to look up in GitHub's search. Google eventually led me to that error being a ticket already, yes, but I've been trying to deploy this particular stack for a LONG time.

Bonus request: alias files like... a repo I'm trying to use utilizes some config files and the '.env' file in the root of the repo directory, but only with '.example' appended to the filenames. You're supposed to normally git clone the repo, 'cp' the files to generate the proper files without the example ending and edit them as you please.

This is complete bonkers of course, would be hella cool if I could tell Portainer to look for the '.env.example' file and use it as '.env' instead. Sadly the creator of the app I'm trying to use doesn't really have more than docker's CLI on their mind.

I doubt this will be the last app of that kind I'll come across. Right now I'm band aiding this by having a personal repo set up for this from which I deploy and intend to switch the repo URL to the official repo later when I got it to deploy because I won't need that initial batch of files anymore. The drawback of my approach vs. just aliasing and overriding the config parts I need changed from within Portainer are that I'll obviously miss out on any official additions to those config files that may be needed later on down the road, not getting new (needed) default settings I may not even need to change.

[chrisbecke]

Is the intention of "Additional Paths" to provide multiple "--compose-file" parameters effectively?
I hope thats what it means.

[deviantony]

@GlassedSilver I'll add proper relative paths support to the list of enhancements for this feature.

Regarding the alias files like request, I'm not sure yet as we can enter the realm of custom naming here and it might be tricky to manage (the .env.example not being a standard).

@chrisbecke yes, that's we're also aiming to provide this ability with our following enhancement.

This feature is of importance to us and we have a lot of enhancements that are gonna follow but they will come incrementally. I'll open a separate issue to track all the enhancements and create a new place of discussion after this one is added.

[deviantony]

Closed via #5139

I'll open a separate issue to track and discuss further enhancements on this feature.

[CurlyFlow]

wow. do i really need 1000 lines of code to send a docker-compose up to portainer? or am i missing something

[deviantony]

@eihns I'm not sure what you're talking about here.

For everyone that is interested on that topic and potential enhancements, I've opened a separate issue: #5223 that we can use to discuss!