"aws: command not found" in local-exec after installing AWS CLI in Atlantis image #1605

camados · 2021-05-27T21:04:05Z

Hi, I'm using a Bridgecrew module that, on apply, executes aws sns commands in a local-exec. The aws cli was not installed into the base Atlantis image, so I went ahead and rebuilt my Atlantis image to include the AWS CLI, as well as updated the PATH and symlinked the aws executable with /bin/aws which local-exec uses. I'm still getting the "local-exec: /bin/aws not found" error. Can you help?

micborens · 2021-06-14T14:32:36Z

I am facing a pretty much similar problem as you but using the Helm and Kubernetes provider.

Error from atlantis plan

Error: Get "https://xxxx.gr7.eu-west-1.eks.amazonaws.com/apis/rbac.authorization.k8s.io/v1/namespaces/grafana/rolebindings/delete-pod-to-platform": getting credentials: exec: executable aws not found

It looks like you are trying to use a client-go credential plugin that is not installed.

To learn more about this feature, consult the documentation available at:
      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins

my provider

  kubernetes {
    host                   = module.finder.this.endpoint
    cluster_ca_certificate = base64decode(module.finder.this.certificate_authority[0].data)
    exec {
      api_version = "client.authentication.k8s.io/v1alpha1"
      args        = ["eks", "get-token", "--cluster-name", module.finder.this.id, "--role-arn", module.accounts.tf_role]
      command     = "aws"
    }
  }
}

terraform registry doc:

https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs#exec-plugins

I work fine on my local because I have awscli installed.

cilindrox · 2021-06-24T19:18:39Z

The following Dockerfile works and has the aws-cli available on $PATH for any commands/local-exec:

FROM runatlantis/atlantis:v0.17.1@sha256:74a1bfd6c863ef194a3bf15a4a1991a4cdc50375110a294599d7e02200a9dbb0

RUN apk --no-cache add aws-cli

micborens · 2021-08-09T09:42:57Z

Hello aws-cli is not the latest version.
from an alpine with awscli installed from apk cmd:

bash-5.1# aws --version
aws-cli/1.18.177 Python/3.8.10 Linux/5.10.25-linuxkit botocore/1.19.47

from an alpine with a manual installation:

bash-5.1# aws --version
aws-cli/2.2.27 Python/3.8.8 Linux/5.10.25-linuxkit exe/x86_64.alpine.3 prompt/off
bash-5.1#

I suggest you to install awscli using the second way (munual)

FROM runatlantis/atlantis:latest

# https://github.com/aws/aws-cli/issues/4685
ENV GLIBC_VER=2.33-r0

RUN curl -sL https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-${GLIBC_VER}.apk \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-bin-${GLIBC_VER}.apk \
    && curl -sL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip \
    && apk add --update --no-cache util-linux openssl python3 glibc-${GLIBC_VER}.apk glibc-bin-${GLIBC_VER}.apk \
    && unzip awscliv2.zip && aws/install \
    && rm -rf awscliv2.zip aws glibc-${GLIBC_VER}.apk glibc-bin-${GLIBC_VER}.apk \
        /usr/local/aws-cli/v2/*/dist/aws_completer \
        /usr/local/aws-cli/v2/*/dist/awscli/data/ac.index \
        /usr/local/aws-cli/v2/*/dist/awscli/examples \
        /var/cache/apk/*

jseiser · 2022-06-29T16:28:21Z

Is it not currently possible to run Atlantis in a dockerfile, while being able to utilize the aws cli for things like kubernetes provider?

Anything I try from the link above, fail to install with

ERROR: glibc-2.35-r0: trying to overwrite lib/ld-linux-x86-64.so.2 owned by gcompat-1.0.0-r4.

ganeshk1928 · 2022-07-18T06:07:25Z

I’m also getting similar output

---> Running in 2eef0326330d
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
(1/2) Installing glibc (2.34-r0)
ERROR: glibc-2.34-r0: trying to overwrite lib/ld-linux-x86-64.so.2 owned by gcompat-1.0.0-r4.
ERROR: glibc-2.34-r0: trying to overwrite lib64/ld-linux-x86-64.so.2 owned by gcompat-1.0.0-r4.
(2/2) Installing glibc-bin (2.34-r0)
Executing glibc-bin-2.34-r0.trigger
1 error; 49 MiB in 48 packages
The command '/bin/sh -c apk add --no-cache glibc-${GLIBC_VER}.apk glibc-bin-${GLIBC_VER}.apk' returned a non-zero code: 1```

ganeshk1928 · 2022-07-18T06:10:34Z

@jseiser any workaround ?

cilindrox · 2022-07-18T19:26:45Z

@ganeshk1928 see this SO answer.

ARG ALPINE_VERSION=3.15.4

FROM alpine:${ALPINE_VERSION}

ARG GLIBC_VERSION=2.34-r0
ARG AWSCLI_VERSION=2.6.1

# install glibc compatibility for alpine
RUN apk --no-cache add \
        binutils \
        curl \
    && curl -sL https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-bin-${GLIBC_VERSION}.apk \
    && curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-i18n-${GLIBC_VERSION}.apk \
    && apk add --no-cache \
        glibc-${GLIBC_VERSION}.apk \
        glibc-bin-${GLIBC_VERSION}.apk \
        glibc-i18n-${GLIBC_VERSION}.apk \
    && /usr/glibc-compat/bin/localedef -i en_US -f UTF-8 en_US.UTF-8 \
    && curl -sL https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${AWSCLI_VERSION}.zip -o awscliv2.zip \
    && unzip awscliv2.zip \
    && aws/install \
    && rm -rf \
        awscliv2.zip \
        aws \
        /usr/local/aws-cli/v2/current/dist/aws_completer \
        /usr/local/aws-cli/v2/current/dist/awscli/data/ac.index \
        /usr/local/aws-cli/v2/current/dist/awscli/examples \
        glibc-*.apk \
    && find /usr/local/aws-cli/v2/current/dist/awscli/botocore/data -name examples-1.json -delete \
    && apk --no-cache del \
        binutils \
        curl \
    && rm -rf /var/cache/apk/*

looks like newer versions of the aws-cli break this glibc shim, but this old version seems enough for configuring the kubectl and helm provider using the new client.authentication.k8s.io/v1beta1 api.

You'll just have to extend FROM the current atlantis image and add the RUN commands from above, or use an init container for Atlantis, etc, and that'll get you a 2.x version of the aws-cli binary.

ganeshk1928 · 2022-07-19T04:41:15Z

@cilindrox Can you share the full dockerfile with extended FROM if possible 😊 I'm unable to do that step 😔

Or

You can edit my dockerfile

# BASE IMAGE
FROM ghcr.io/runatlantis/atlantis:v0.19.6

# COPY ID_RSA KEY TO IMAGE
COPY id_rsa /home/atlantis/.ssh/id_rsa
RUN chmod 400 /home/atlantis/.ssh/id_rsa
RUN ssh-keyscan github.com > /home/atlantis/.ssh/known_hosts 
RUN chown -R atlantis:atlantis /home/atlantis/.ssh/

#INSTALLING JQ, CURL AND AWS CLI
ENV GLIBC_VER=2.34-r0
RUN apk --no-cache add binutils curl jq sudo
RUN curl -sL https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub
RUN curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-${GLIBC_VER}.apk
RUN curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VER}/glibc-bin-${GLIBC_VER}.apk
RUN apk add --no-cache glibc-${GLIBC_VER}.apk glibc-bin-${GLIBC_VER}.apk
RUN curl -sL https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o awscliv2.zip && unzip awscliv2.zip && aws/install && rm -rf awscliv2.zip aws /usr/local/aws-cli/v2/*/dist/aws_completer /usr/local/aws-cli/v2/*/dist/awscli/data/ac.index /usr/local/aws-cli/v2/*/dist/awscli/examples
RUN rm glibc-${GLIBC_VER}.apk && rm glibc-bin-${GLIBC_VER}.apk
RUN rm -rf /var/cache/apk/*

#ADD ATLANTIS USER TO SUDO GROUP AND PROVIDE SUDO PREVILIGES
RUN echo '%wheel ALL=(ALL) ALL' > /etc/sudoers.d/wheel
RUN adduser atlantis wheel
RUN echo "atlantis ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

USER atlantis

cilindrox · 2022-07-19T12:05:18Z

Sure, here's an example extending from the atlantis base:

FROM ghcr.io/runatlantis/atlantis:v0.19.6@sha256:2d1efea6ae65b075a29df211809f27d8239e77e75a6fa6222778cf04f4119aef

ARG GLIBC_VERSION=2.34-r0
ARG AWSCLI_VERSION=2.6.1

# gcompat (required and installed by atlantis) triggers a conflict when
# installing the glibc shim.
RUN apk --no-cache add \
		binutils \
	&& curl -sL https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub -o /etc/apk/keys/sgerrand.rsa.pub \
	&& curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk \
	&& curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-bin-${GLIBC_VERSION}.apk \
	&& curl -sLO https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-i18n-${GLIBC_VERSION}.apk \
	&& apk del gcompat \
	&& apk add --no-cache \
			glibc-${GLIBC_VERSION}.apk \
			glibc-bin-${GLIBC_VERSION}.apk \
			glibc-i18n-${GLIBC_VERSION}.apk \
	&& /usr/glibc-compat/bin/localedef -i en_US -f UTF-8 en_US.UTF-8 \
	&& curl -sL https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${AWSCLI_VERSION}.zip -o awscliv2.zip \
	&& unzip awscliv2.zip \
	&& aws/install \
	&& rm -rf \
			awscliv2.zip \
			aws \
			/usr/local/aws-cli/v2/current/dist/aws_completer \
			/usr/local/aws-cli/v2/current/dist/awscli/data/ac.index \
			/usr/local/aws-cli/v2/current/dist/awscli/examples \
			glibc-*.apk \
	&& find /usr/local/aws-cli/v2/current/dist/awscli/botocore/data -name examples-1.json -delete \
	&& apk --no-cache del \
			binutils \
	&& rm -rf /var/cache/apk/*

Keep in mind I'm kinda playing with fire here, but it looks like gcompat was added for convenience on #2175 and it's not mission-critical. However, you might run into issues with some providers or custom workflows that depend on other binaries that aren't compiled against musl.

ganeshk1928 · 2022-07-19T14:21:37Z

Thank you very much @cilindrox

nitrocode · 2022-07-28T00:01:27Z

If you're using arm osx m1, then you need to run the docker build with --platform linux/arm64 or it will build but then aws --version will not work

This works

$ docker build --platform linux/amd64 -t atlantis-test .
$ docker run -it atlantis-test aws --version
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
aws-cli/2.7.19 Python/3.9.11 Linux/5.10.104-linuxkit exe/x86_64.alpine.3 prompt/off

This fails

$ docker build -t atlantis-test-arm .
$ docker run -it atlantis-test-arm aws --version
aws: error while loading shared libraries: libz.so.1: cannot open shared object file: No such file or directory

gaspo53 · 2023-03-30T21:04:42Z

For those wanting aws-cli: https://hub.docker.com/r/chatwork/atlantis-aws

nitrocode · 2023-03-31T00:40:55Z

I wouldn't use a docker image you find on dockerhub unless it's official. It's highly encouraged to create a private Atlantis image for your organization so you can customize the necessary binaries that you workflows need.

It might be as simple as this

FROM ghcr.io/runatlantis/atlantis:v0.23.4

RUN apk add aws-cli

camados changed the title ~~Unable to use AWS CLI commands in local-exec~~ Unable to use AWS CLI commands in local-exec after installing AWS CLI in Atlantis image May 27, 2021

camados changed the title ~~Unable to use AWS CLI commands in local-exec after installing AWS CLI in Atlantis image~~ "aws: command not found" in local-exec after installing AWS CLI in Atlantis image May 27, 2021

jamengual closed this as completed Aug 26, 2022

nitrocode mentioned this issue Sep 14, 2022

Support flavored Docker images i.e. awscli v2, tfsec, or alpine, debian #2523

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

"aws: command not found" in local-exec after installing AWS CLI in Atlantis image #1605

"aws: command not found" in local-exec after installing AWS CLI in Atlantis image #1605

camados commented May 27, 2021

micborens commented Jun 14, 2021 •

edited

cilindrox commented Jun 24, 2021

micborens commented Aug 9, 2021

jseiser commented Jun 29, 2022

ganeshk1928 commented Jul 18, 2022 •

edited

ganeshk1928 commented Jul 18, 2022

cilindrox commented Jul 18, 2022 •

edited

ganeshk1928 commented Jul 19, 2022 •

edited

cilindrox commented Jul 19, 2022 •

edited

ganeshk1928 commented Jul 19, 2022

nitrocode commented Jul 28, 2022

gaspo53 commented Mar 30, 2023

nitrocode commented Mar 31, 2023 •

edited

"aws: command not found" in local-exec after installing AWS CLI in Atlantis image #1605

"aws: command not found" in local-exec after installing AWS CLI in Atlantis image #1605

Comments

camados commented May 27, 2021

micborens commented Jun 14, 2021 • edited

cilindrox commented Jun 24, 2021

micborens commented Aug 9, 2021

jseiser commented Jun 29, 2022

ganeshk1928 commented Jul 18, 2022 • edited

ganeshk1928 commented Jul 18, 2022

cilindrox commented Jul 18, 2022 • edited

ganeshk1928 commented Jul 19, 2022 • edited

cilindrox commented Jul 19, 2022 • edited

ganeshk1928 commented Jul 19, 2022

nitrocode commented Jul 28, 2022

gaspo53 commented Mar 30, 2023

nitrocode commented Mar 31, 2023 • edited

micborens commented Jun 14, 2021 •

edited

ganeshk1928 commented Jul 18, 2022 •

edited

cilindrox commented Jul 18, 2022 •

edited

ganeshk1928 commented Jul 19, 2022 •

edited

cilindrox commented Jul 19, 2022 •

edited

nitrocode commented Mar 31, 2023 •

edited