[QUESTION] Usage with AWS ecr registry

[posquit0]

Hi, I'm suffering from configuring ECR registry in portainer.
I'm using EC2 instance with IAM role which has an access policy to ECR.
And installed amazon ecr credential helper in every docker nodes.
https://github.com/awslabs/amazon-ecr-credential-helper

Portainer app live on swarm master node.
I registered ecr. However, portainer always fail with ecr images.

How can I do? Is there no support for ECR?

[posquit0]

I need something like this.
https://github.com/rancher/rancher-ecr-credentials

[WTFKr0]

Portainer only support auth by login/password on registry
The rancher ecr credential project is created to enable this feature on rancher, by periodically update credentials
If you want to use ecr images in portainer, we need to have similar behavior

[deviantony]

Yes, I believe that we will need to integrate a feature similar to the rancher-ecr-credentials project to automatically update credentials.

Access to an ECR registry is controlled by AWS IAM, an IAM user must request temporary credentials to the registry using the AWS API, see http://rancher.com/using-amazon-container-registry-service/ for more details.

[Mobe91]

I would like to use portainer to manage my swarm cluster running on AWS. However, I need portainer to be able to access my Docker images stored in ECR. Without such a feature, I cannot use portainer. :(

[tcjennings]

I have a similar use case where I launch a swarm in AWS and use ECR for the images (and CodeCommit as repository, which is relevant to a different issue). I have applied an IAM role to my instances which grants Pull permissions to the repositories (i.e., I do not have to run aws ecr get-login or docker login). My workflow for launching stacks was too manual: ssh to the manager node, execute a shell script. Yet, registry auth worked.

I have previously used https://github.com/dockersamples/docker-swarm-visualizer as a "GUI" but want to use portainer as a management layer for my stacks. Since I run portainer as a service on my swarm, it necessarily runs as a container itself, which puts the burden of the registry authentication inside the container, and what works on the host doesn't work in the container.

My workaround is this: wrap the portainer docker image in my own Dockerfile and use my own image instead:

Dockerfile

FROM golang:1.9 as ecr-helper
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login
WORKDIR /go/src/github.com/awslabs/amazon-ecr-credential-helper
RUN make

FROM busybox as busybox
RUN which busybox

FROM portainer/portainer:latest
ENV HOME=/
COPY --from=ecr-helper /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/docker-credential-ecr-login /bin/docker-credential-ecr-login
COPY config.json /.docker/config.json
COPY --from=busybox /bin/busybox /bin/busybox
RUN ["/bin/busybox","--install","-s","/bin"]

This gives me a portainer image that's a little bigger but contains the AWS ecr-login helper binary (and a busybox for good measure so I can get a shell and poke around). Given the HOME environment and the config.json (straight from the ecr helper README, see below), I am able to launch stacks from portainer with images from the ECR.

config.json

{
	"credsStore": "ecr-login"
}

Your mileage may vary, and I do +1 a feature that would bring support for revolving ecr credentials into portainer, but with IAM roles my workaround might be useful to others.

[jgrasl]

@tcjennings @deviantony unfortunately after the update to 1.17.0 aws login doesn't work anymore with the ecr login helper. If I'll have time I'll check what is changed why the binary is not called when docker pull initiated. I rebuilt the same with 1.16.5 which works perfectly but with the latest version ecr cache never get filled and as I see the helper binary is not called when a stack deployment ran.

[deviantony]

@jgrasl starting with 1.17 Portainer is using a specifc config.json file located in /config.json.

If you're using the Dockerfile example above, then changing this line might solve your issue:

COPY config.json /config.json

Note that the location of the config.json might be updated after solving #1888

[jgrasl]

@deviantony As I see from the changes in the release the config json file is always overwritten by main() - > initStackManager -> NewStackManager - > updateDockerCLIConfiguration. It works fine if I overwrite the HTTPheaders in the json file after portainer process is started but I cannot overwrite it at build time.
I'll take a look if I can start portainer with a script inside the container instead of with the /portainer entry point so then I can modify the content of the json file after the process was initiated. But I fear the HTTP headers information are stored for a reason so I should not really remove it from that file. I'm not using the agent currently but it can be a problem for someone else.

[deviantony]

Right, we'll try to preserve the content of the config.json file when solving #1888

[deviantony]

@jgrasl keen to give a try with portainer/portainer:pr1898 ?

[jgrasl]

@deviantony With the pr1898 the content of the /config.json is preserved and a new config.json is created under /data. Altough that new config.json file only contains the HTTP header json nothing from the file copied build time and so the ecr login is not working with the cred helper.

[deviantony]

@jgrasl well with pr1898 you'd need to update your Dockerfile:

COPY config.json /data/config.json

[jgrasl]

@deviantony sorry I forgot that. Now I did change it unfortunately now the data from my config json is not preserved. Under /data/config.json is only the HTTP headers. I am mapping the data volume to an external volume trough docker volumes to persist portainer config, can that be a problem?

[deviantony]

@jgrasl you're using FROM portainer/portainer:pr1898 in your Dockerfile right? Any existing config.json in that volume?

[jgrasl]

@deviantony yes using the pr docker image

FROM golang:1.9 as ecr-helper                                                                                            
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login                  
WORKDIR /go/src/github.com/awslabs/amazon-ecr-credential-helper                                                          
RUN make                                                                                                                 
                                                                                                                         
FROM busybox as busybox                                                                                                  
RUN which busybox                                                                                                        
                                                                                                                         
FROM portainer/portainer:pr1898                                                                                          
COPY --from=ecr-helper /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/docker-credential-ecr-login /bin/docker-credential-ecr-login
COPY config.json /data/config.json                                                                                       
COPY --from=busybox /bin/busybox /bin/busybox                                                                            
RUN ["/bin/busybox","--install","-s","/bin"]

Before starting up the container I deleted the config.json file from the mounted data volume so should be no json file there at startup.

[deviantony]

Actually, mounting a volume to persist Portainer data will map a folder on disk to /data, making that COPY config.json /data/config.json statement useless.

I'm trying to think of a way to persist user changes to config.json before deployment.

[deviantony]

@jgrasl How are you persisting Portainer data? Using a named volume? If you're using a named volume AND the volume is empty (e.g. just created) it will have the config.json copied from the Dockerfile. If you're reusing an existing data volume, I believe the config.json file will not be copied.

Might be problematic for people upgrading their Portainer instance, still searching for a better way to keep config.json changes.

[jgrasl]

@deviantony no, just using a mapped directory with docker run, but it can be named of course if it's needed.
-v /portainer/data/path/on/server:/data

[jgrasl]

@deviantony for now I just put the config.json file under the mapped directory with the credsStore setup and startup portainer. This way my config got merged with the HTTP headers and ecr login works fine with the pr1898.

[deviantony]

@jgrasl this is what I'm thinking as well. If anybody using this workaround wants to continue using it and have existing Portainer data, they'll have to put their config.json in the root of the existing data directory. If people are starting to use this workaround and do not have any existing data, then the config.json will be created in their data directory.

Nonetheless we'll need to find a real solution for this later on.

[jgrasl]

@deviantony I don't know if you plan to do any other support for ecr login but I think the credential helper could be a final solution. It only needs the credentials file which is already used by everyone who uses the aws cli so if the credsStore in the config json could be included by default in the portainer image we wouldn't need to put he file manually under the persisted /data directory only provide the credentials file.
But I'm completely happy with the current solution too since it works fine for me.
Thanks for the fix!

[deviantony]

We have not thought about it yet, quite busy with other topics atm :-)

[jgrasl]

@deviantony Of course I understand, busy release time:) I can prepare a PR with what I thought and you guys can decide later if you'd like to merge it for a later release or not.

[reineruhry]

I've tried all workarounds with the new 1.17.0, may be because I'm using agents, it's not working! I tried adding config.json on /, on data on .docker...
Any idea on how have it working?

[xgenvn]

Full working example as I have tried recently:

$ ls .

Dockerfile
config.json

• Dockerfile

FROM golang:1.9 as ecr-helper
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login
WORKDIR /go/src/github.com/awslabs/amazon-ecr-credential-helper
RUN make

FROM busybox as busybox
RUN which busybox

FROM portainer/portainer:1.24.1
ENV HOME=/
COPY --from=busybox /bin/busybox /bin/busybox
RUN ["/bin/busybox","--install","-s","/bin"]
COPY --from=ecr-helper /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/docker-credential-ecr-login /bin/docker-credential-ecr-login
COPY config.json /.docker/config.json

• config.json

{
    "credsStore": "ecr-login"
}

• Build the portainer custom image

docker build -t portainer-custom:latest .

• Deploy portainer with the environment:

version: "3.3"

services:
  portainer:
    image: portainer-custom:latest
    environment: 
      AWS_ACCESS_KEY_ID:
      AWS_SECRET_ACCESS_KEY:

• Big note You CANNOT pull from portainer UI. But you can actually deploy stack which will be able to pull the images from. AWS ECR.
• In case you want to test out, enter portainer console under Containers > portainer container:

/ # ./docker pull <ECR Repo/image>

However, I really expect that Portainer can support this feature, perhaps pulling using remote command or using a custom container (just like above but not portainer specific).

[dbartokthomas]

Full working example as I have tried recently:

$ ls .

Dockerfile
config.json

• Dockerfile

FROM golang:1.9 as ecr-helper
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login
WORKDIR /go/src/github.com/awslabs/amazon-ecr-credential-helper
RUN make

FROM busybox as busybox
RUN which busybox

FROM portainer/portainer:1.24.1
ENV HOME=/
COPY --from=busybox /bin/busybox /bin/busybox
RUN ["/bin/busybox","--install","-s","/bin"]
COPY --from=ecr-helper /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/docker-credential-ecr-login /bin/docker-credential-ecr-login
COPY config.json /.docker/config.json

• config.json

{
    "credsStore": "ecr-login"
}

• Build the portainer custom image

docker build -t portainer-custom:latest .

• Deploy portainer with the environment:

version: "3.3"

services:
  portainer:
    image: portainer-custom:latest
    environment: 
      AWS_ACCESS_KEY_ID:
      AWS_SECRET_ACCESS_KEY:

• Big note You CANNOT pull from portainer UI. But you can actually deploy stack which will be able to pull the images from. AWS ECR.
• In case you want to test out, enter portainer console under Containers > portainer container:

/ # ./docker pull <ECR Repo/image>

However, I really expect that Portainer can support this feature, perhaps pulling using remote command or using a custom container (just like above but not portainer specific).

Have you tried this with Portainer 2.0?

[garciabruno]

Full working example as I have tried recently:

$ ls .

Dockerfile
config.json

• Dockerfile

FROM golang:1.9 as ecr-helper
RUN go get -u github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login
WORKDIR /go/src/github.com/awslabs/amazon-ecr-credential-helper
RUN make

FROM busybox as busybox
RUN which busybox

FROM portainer/portainer:1.24.1
ENV HOME=/
COPY --from=busybox /bin/busybox /bin/busybox
RUN ["/bin/busybox","--install","-s","/bin"]
COPY --from=ecr-helper /go/src/github.com/awslabs/amazon-ecr-credential-helper/bin/local/docker-credential-ecr-login /bin/docker-credential-ecr-login
COPY config.json /.docker/config.json

• config.json

{
    "credsStore": "ecr-login"
}

• Build the portainer custom image

docker build -t portainer-custom:latest .

• Deploy portainer with the environment:

version: "3.3"

services:
  portainer:
    image: portainer-custom:latest
    environment: 
      AWS_ACCESS_KEY_ID:
      AWS_SECRET_ACCESS_KEY:

• Big note You CANNOT pull from portainer UI. But you can actually deploy stack which will be able to pull the images from. AWS ECR.
• In case you want to test out, enter portainer console under Containers > portainer container:

/ # ./docker pull <ECR Repo/image>

However, I really expect that Portainer can support this feature, perhaps pulling using remote command or using a custom container (just like above but not portainer specific).

Have you tried this with Portainer 2.0?

I've tried it with Portainer's latest version and works just fine. I managed to adapt it and made a little repo using @xgenvn 's answer

https://github.com/filadd/ecr_portainer

[mek97]

We can use httpd server to expose an api to execute ./docker pull <ECR Repo/image>. (Helped me with CI/CD)
Implementation - https://github.com/mek97/portainer-ecr

[stuartaccent]

Am new to portainer firstly tnx its awesome.

Another way that just worked for me is to add the registry in the usual manner with username of AWS and password as the output of aws ecr get-login-password. I know it will need updating but it pulls successfully in the first instance.

Using latest version of portainer-ce.

[deviantony]

We're planning to add support for AWS ECR registry later this year, this is currently earmarked around Q2 2021.

[huib-portainer]

Question for all the AWS ECR users here.
Do you have any scenarios where you are forced to use MFA when connecting to ECR from Portainer?

By adding an AWS user with programmatic access that doesn't use MFA, all we need to pull from ECR is an AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, region and the URL of the registry.

But as soon as two factor authentication is enforced we'll need the ARN of the MFA device as well as a code (e.g. from the authenticator app). So I'm trying to figure out what use cases, if any, there are for this scenario.

[xgenvn]

@huib-portainer at least no for me since that's overcomplicated my workflow. Imagine the auto-deployment through webhook scenario, I don't think we enter the code to authenticate a good idea.

[huib-portainer]

Thanks for that!

This is what we're currently thinking:

[Duduzera1997]

We're planning to add support for AWS ECR registry later this year, this is currently earmarked around Q2 2021.

u have a new prevision?

[huib-portainer]

It's actually still in the pipeline for this year.
Once there's a preview version available, I'll let you know.

[mkostic]

We are planning to switch to Portainer. Any news about this?

[huib-portainer]

It's still on the roadmap for this year.

[honeycomb-cheesecake]

Hi, currently we use amazon-ecr-credential-helper to pull images down to the local machine and then use Portainer's "Deploy Stack" on our running stack to update the image. Obviously it would be nice if it was all under one roof (i.e. Portainer) but it "works" for us right now. When it is stated as "on the roadmap for this year" to pull from ECR, is there an ETA as we are in November now. Asking to determine whether it's worth rolling our own, or even potentially offering to help test if that's even possible.

[huib-portainer]

Hi, it's currently being worked on, so we're still hoping before the end of the year, but there's currently no PR for it so nothing for you to try out just yet.

[honeycomb-cheesecake]

Hi, any update on this one? Is it still on the roadmap for this year or has it been pushed to next year?

[huib-portainer]

You can give it a try by using the image portainerci/portainer:pr6165.
Please let us know how that's working for you.
Note that this is a development build and should not be used in a production environment.

[honeycomb-cheesecake]

Hi @huib-portainer, thanks for this can't wait to try it out!!! 🥳🥳🥳