Retrieve OIDC provider thumbprint not populating which is required for EKS service accounts #10104

marcincuber · 2019-09-13T17:18:00Z

Description

New or Affected Resource(s)

Currently I can specify the following:

resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = []
  url             = aws_eks_cluster.cluster.identity.0.oidc.0.issuer
}

There is no way to retrieve thumbprint for that OIDC provider using terraform.

Note that if you create the same OIDC provider in the console, it will automatically populate the thumbprint which is required for EKS service accounts to assume correct IAM Role.

References

Current way of getting thumbprint is documented here -> https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html#thumbstep2

The text was updated successfully, but these errors were encountered:

gkrizek · 2019-09-18T21:47:20Z

+1

I went through the example here:
https://www.terraform.io/docs/providers/aws/r/eks_cluster.html#enabling-iam-roles-for-service-accounts

In which thumbprint_list is an empty list. Everything worked until I tried to actually do an AWS operation:

An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint

I really wish AWS offered an easier way to get the thumbprint of the certificate.

gkrizek · 2019-09-19T03:29:43Z

After some searching it looks like this could solve our issue if implemented:
hashicorp/terraform-provider-tls#52

zzh8829 · 2019-09-24T09:05:24Z

In the mean time here is a quick hack to get around it, not ideal but tested working

echo | openssl s_client -connect oidc.eks.us-west-2.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}'

locals {
  eks-oidc-thumbprint = "$OUTPUT_FROM_ABOVE"
}

resource "aws_iam_openid_connect_provider" "eks-oidc" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [local.eks-oidc-thumbprint]
  url             = "${aws_eks_cluster.cluster.identity.0.oidc.0.issuer}"
}

marcincuber · 2019-09-24T16:38:52Z

There is an automated way that I received from kubernetes slack which is using external source:

### External cli kubergrunt
data "external" "thumb" {
  program = ["kubergrunt", "eks", "oidc-thumbprint", "--issuer-url", aws_eks_cluster.cluster.identity.0.oidc.0.issuer]
}
### OIDC config
resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumb.result.thumbprint]
  url             = aws_eks_cluster.cluster.identity.0.oidc.0.issuer
}

zzh8829 · 2019-09-24T18:58:37Z

I personally think this thumbprint thing should live in terraform-provider-aws repo.
SHA1 finger print are usually in the form of

SHA1 Fingerprint=63:96:61:30:76:16:08:20:97:18:C5:04:5C:FF:B4:85:6F:B5:39:76

But aws_iam_openid_connect_provider needs it to be like (yes lowercase)

63966130761608209718c5045cffb4856fb53976

I don't know if this specific transform is general enough to live in Terraform TLS provider.

dogzzdogzz · 2019-09-25T09:31:52Z

For anyone prefer not to install kubergrunt, I merge both @zzh8829 and @marcincuber workaround to make terraform automatically retrieve thumbprint from external script.

thumbprint.sh

#!/bin/bash

THUMBPRINT=$(echo | openssl s_client -connect oidc.eks.$1.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

terraform.tf

data "external" "thumbprint" {
  program = ["thumbprint.sh", data.aws_region.current.name]
}

resource "aws_iam_openid_connect_provider" "this" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumbprint.result.thumbprint]
  url             = "${aws_eks_cluster.this.identity.0.oidc.0.issuer}"
}

matteosilv · 2019-09-30T13:56:44Z

@dogzzdogzz thanks for your solution! Unfortunately i tried it without any luck. It is giving me an incorrect thumbprint.

From my understanding of the aws tutorial we need to extract only the last certificate between BEGIN CERTIFICATE and END CERTIFICATE, that one is the certificate of the root CA.

Then we can give it to openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}' to extract the thumbprint

jmtsi · 2019-09-30T19:26:46Z

Yep, that produces a non-working thumbprint. Here's my horrible oneliner to produce a working thumbprint:
echo | openssl s_client -servername oidc.eks.${REGION}.amazonaws.com -showcerts -connect oidc.eks.${REGION}.amazonaws.com:443 2>&- | tail -r | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tail -r | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}'

Edit: this is for OSX, for GNU/Linux replace tail -r with tac

matteosilv · 2019-10-01T21:13:39Z

Thanks! I combined the two solutions to get a working terraform resource

chiefy · 2019-10-11T13:55:33Z

So the solutions above don't work for our setup, we use an https proxy and thus the CA chain won't be correct if I got through that. I can tunnel through a bastion, but am I missing something here, that CA root will be mostly static for each AWS region, no? Wouldn't it be viable to just keep the fingerprints for each region in a TF map until a better solution presents itself?

willthames · 2019-10-23T04:47:42Z

@chiefy you don't even need a map - it looks like the root CA is the same for the two regions I tried, and I imagine that's the case for all the regions.

Setting thumbprints to ["9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"] should suffice :)

chiefy · 2019-10-23T20:43:36Z

@willthames thanks for confirming my sanity.

dayglojesus · 2019-10-30T20:46:40Z

For anybody who is running into issues with differing or inconsistent thumbprints, this might help ...

~~When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the cluster (from one of your EKS workers), you get a cert like:~~
When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the pod, you get a cert like:

subject=CN = oidc.eks.us-west-2.amazonaws.com
issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

☝️ This is the thumbprint you want.

But, if you are performing this same operation from outside the cluster, you will likely receive something like:

subject=/CN=*.execute-api.us-west-2.amazonaws.com
issuer=/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon

HTH

Grejeru · 2019-10-31T07:22:23Z

@dayglojesus yes, it helps, but still there is a problem with automating this. If we need to run command from inside Pod to get thumbprint required to automate IAM role policies for ServiceAccounts makes it impossible to work :-/
Nevertheless thx, as it solve my issue why script generated thumbprint didn't worked and had to find it other way. Which is even funnier, there are more then 1 cert, to be exact 4 of them and with this list I got it working.

dayglojesus · 2019-10-31T15:42:26Z

@Grejeru I agree, this shouldn't be a hard requirement, but per @chiefy's comments ... You should be able to construct a map of the correct thumbs, one for each required region.

rastakajakwanna · 2019-11-07T13:30:15Z

Setting thumbprints to ["9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"] should suffice :)

@willthames well, this will result in constant drift because the thumbprint needs to be lowercase (at least that's my experience).
However, all our eu regions have the same thumbprint. I tried the script from outside of VPC as well as from Gitlab runner in EKS and I've got the same results/thumbprint.

I merged my previous ugly one-liner with the suggestions here and made this (for tf v.0.11.x):

<module_name>/bin/get_thumbprint.sh

#!/bin/bash
set -e 

XR="${1:-eu-west-1}"
XT=`mktemp`
XXT=`mktemp`

function cleanup {
  rm -f ${XT} ${XXT}
}

trap cleanup SIGHUP SIGINT SIGTERM EXIT

THUMBPRINT=$(echo QUIT | openssl s_client -showcerts -connect oidc.eks.${XR}.amazonaws.com:443 2>/dev/null > ${XT}; cat ${XT} | sed -n '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' | tac | awk '/-----BEGIN CERTIFICATE-----/ {exit} 1' > ${XXT} && echo '-----BEGIN CERTIFICATE-----' >> ${XXT} && tac ${XXT} > ${XT}; openssl x509 -in ${XT} -fingerprint -noout | sed -r 's|.*+?=(.*)|\1|g' | sed 's|:||g' | awk '{print tolower($1)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

<module_name>/data.tf

data "aws_region" "current" {}
data "external" "thumbprint" {
  program = ["${path.module}/bin/get_thumbprint.sh", "${data.aws_region.current.name}"]
}

<module_name>/main.tf

resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["${data.external.thumbprint.result.thumbprint}"]
  url             = "${data.aws_eks_cluster.this.identity.0.oidc.0.issuer}"
}

It seems that this works perfectly fine for the kube-system aws-node service account. At least as much as I can say after two days....

willthames · 2019-11-07T14:51:27Z

@rastakajakwanna yes, you're quite right, I think I wrote my comment after realising that it worked, but before realising about the drift you mention.

But I'm literally just hardcoding the (lowercase) value into a variable. I don't see the benefit of having terraform run a script to generate a value that so far is consistent across all regions, and given it's a root CA, likely will remain so for a decade.

mzupan · 2019-12-13T18:10:27Z

This works with terraform cloud

#!/bin/bash

THUMBPRINT=$(echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | tac | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tac | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

data "external" "thumbprint" {
  program = [format("%s/bin/get_thumbprint.sh", path.module), data.aws_region.current.name]
}

resource "aws_iam_openid_connect_provider" "this" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumbprint.result.thumbprint]
  url             = module.eks.cluster_oidc_issuer_url
}

jujugrrr · 2020-01-03T16:15:44Z

Should this issue be marked as a bug? The terraform provider documentation is misleading, as an empty thumbprint doesn't allow to use EKS IAM with service account.

marcincuber · 2020-01-03T16:24:05Z

@jujugrrr I don't think this is a bug. Terraform is simply missing a way to retrieve the thumbprint. I do agree that documentation is not perfect.

jujugrrr · 2020-01-03T16:29:18Z

@jujugrrr I don't think this is a bug. Terraform is simply missing a way to retrieve the thumbprint. I do agree that documentation is not perfect.

Fair enough 😄 , I've raised it with AWS, but it feels it's a sane behavior not to expose the thumbprint through the API response on their side, which is probably why the AWS Console auto populate it for convenience.

Let's follow #10217

cippaciong · 2020-01-16T17:37:32Z

If anyone needs to run @mzupan example on macOS/BSD/others, please consider that tac is not available there by default and you need to either install coreutils or replace it with tail -r, which is what I did:

#/bin/bash

THUMBPRINT=$(echo | \
    openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | \
    tail -r | \
    sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | \
    tail -r | \
    openssl x509 -fingerprint -noout | \
    sed 's/://g' | awk -F= '{print tolower($2)}')

THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"

echo $THUMBPRINT_JSON

Thanks @mzupan ❤️

s0enke · 2020-01-23T13:20:09Z

~~When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the cluster (from one of your EKS workers), you get a cert like:~~
When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the pod, you get a cert like:

@dayglojesus @Grejeru I think I found the root issue for different certificates from different clients:

It depends on the implementation/version of the openssl command. In OpenSSL (since 1.1?) the connect will automatically send an servername with the connection (Server Name Indication). See [1]

Other implementations like LibreSSL (on Mac) or CodeBuild/AmazonLinux 2 (which use OpenSSL 1.0.2) don't automatically set the servername so we have to provide it.

The following command should work independently from OpenSSL versions/implemenentations:

openssl s_client -connect oidc.eks.eu-central-1.amazonaws.com:443 -servername oidc.eks.eu-central-1.amazonaws.com

[1] https://www.openssl.org/docs/man1.1.1/man1/openssl-s_client.html

mzupan · 2020-01-23T13:38:53Z

@cippaciong

I have an updated version that tests for a env var used by tf cloud

#!/bin/bash



if [[ -z "${ATLAS_WORKSPACE_SLUG}" ]]; then
  APP="tail -r"
else
  APP="tac"
fi

THUMBPRINT=$(echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | ${APP} | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | ${APP} | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

christian-marie · 2020-03-03T07:28:43Z

If your openssl's x509 fingerprint behaviour defaults to something other than sha1 you may get invalid fingerprints:

Member must satisfy constraint: [Member must have length less than or equal to 40, Member must have length greater than or equal to 40]

Being explicit fixes this:

#!/bin/sh
# https://github.com/terraform-providers/terraform-provider-aws/issues/10104

THUMBPRINT=$(echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | tac | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tac | openssl x509 -fingerprint -sha1 -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

groodt · 2020-03-05T22:56:15Z

The thumbprint even seems to be the same cn-northwest-1

echo | openssl s_client -servername oidc.eks.cn-northwest-1.amazonaws.com.cn -showcerts -connect oidc.eks.cn-northwest-1.amazonaws.com.cn:443 2>&- | tail -r | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tail -r | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}'

This adds a list of server certificate thumbprints as a computed attribute to both the aws_eks_cluster resource and aws_eks_cluster data source. It implements the instructions from https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html in go. The happy path acceptance tests have been updated. Fixes hashicorp#10104

scardena · 2020-05-21T19:51:17Z

For anyone prefer not to install kubergrunt, I merge both @zzh8829 and @marcincuber workaround to make terraform automatically retrieve thumbprint from external script.

thumbprint.sh

#!/bin/bash

THUMBPRINT=$(echo | openssl s_client -connect oidc.eks.$1.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

terraform.tf

data "external" "thumbprint" {
  program = ["thumbprint.sh", data.aws_region.current.name]
}

resource "aws_iam_openid_connect_provider" "this" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumbprint.result.thumbprint]
  url             = "${aws_eks_cluster.this.identity.0.oidc.0.issuer}"
}

@mzupan I had to modify this answer to work for me, as I am using it in a module:
program = ["${path.module}/thumbprint.sh", var.aws_region]

Grejeru · 2020-05-22T05:27:54Z

@mzupan I had to modify this answer to work for me, as I am using it in a module:
program = ["${path.module}/thumbprint.sh", var.aws_region]

You can use data "aws_region" "current" {} in the module too, it assumes the region in which Your provider works, don't need to pass aws_region into module then.

romaryoricardo · 2020-05-23T20:11:50Z

You can generate fingerprint and set in your thumbprint_list follow how to this link: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html

Those are all CA's fingerprint by region:

eu-north-1
3ACE16CA6BAE7B16AAE3707096D1DE7D29093AD8
ap-south-1
59ADE3A3A6039BBA092E920FEE413466493F409C
eu-west-3
3FB881BCACBD420928168C739B07EEF47555946B
eu-west-2
7CA6BE9F14E20973CB2C58452DA9B1E2BEB7236B
eu-west-1
CAB073498D7558FEC3B2C414C006ACBA30805431
ap-northeast-2
2EAFC197C15CDEE5426BFD4D27D3321A685F3B78
ap-northeast-1
B715DC079832DA5FC1D4706515BE48BE79A1C871
sa-east-1
CB454452665937052981CA118417B7A162A25F54
ca-central-1
1C8B5245E80A6B7A0E8BF5FFDAB032273D7D5DF1
ap-southeast-1
F719C49FEA86549E159818880E392C1570C953B6
ap-southeast-2
0148872FA02F3A7D6B38AA88FA5397228B28E08B
eu-central-1
9884072430220E6253011B88F940E4F20F53D0CC
us-east-1
598ADECB9A3E6CC70AA53D64BD5EE4704300382A
us-east-2
750B948515281953BC6F3D717A1E1654ECBFA852
us-west-1
89BABC6D46502653516CC0BA38B14A2B7864D161
us-west-2
63966130761608209718C5045CFFB4856FB53976

You can define your variable like this:

thumbprint_list = ["3ACE16CA6BAE7B16AAE3707096D1DE7D29093AD8","59ADE3A3A6039BBA092E920FEE413466493F409C","3FB881BCACBD420928168C739B07EEF47555946B","7CA6BE9F14E20973CB2C58452DA9B1E2BEB7236B","CAB073498D7558FEC3B2C414C006ACBA30805431","2EAFC197C15CDEE5426BFD4D27D3321A685F3B78","B715DC079832DA5FC1D4706515BE48BE79A1C871","CB454452665937052981CA118417B7A162A25F54","1C8B5245E80A6B7A0E8BF5FFDAB032273D7D5DF1","F719C49FEA86549E159818880E392C1570C953B6","0148872FA02F3A7D6B38AA88FA5397228B28E08B","9884072430220E6253011B88F940E4F20F53D0CC","598ADECB9A3E6CC70AA53D64BD5EE4704300382A","750B948515281953BC6F3D717A1E1654ECBFA852","89BABC6D46502653516CC0BA38B14A2B7864D161","63966130761608209718C5045CFFB4856FB53976"]

command for get fingerprint :

echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print $2}'

scardena · 2020-05-25T19:04:52Z

@romaryoricardo I haven't test this yet, but I recommend @mzupan (or @cippaciong for macos) workaround over yours. I am not sure why this happens, but the fingerprints generated using your command are not the same as the ones generated by @mzupan commands (I am going to throw the guess that multiple certificates are being spit out for each command, but only 1 is processed, in @mzupan case is the most recent one, and in your case is the oldest one, although i can be completely wrong). It doesn't mean your solutions shouldn't work, but if we inspect the certificate of the fingerprint that your command spits, we can see that this certificate is about to expire (check the validity, it expires september 2020):

echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -fingerprint -noout -text

whereas in the other, it expires on 2034.
More over, I ran @mzupan script for all regions, and the output fingerprint seems to be the same for absolutely all regions (as @willthames suggests), so in my case, we decided to simply hardcode the fingerprint in the terraform module, although we still have the workaround commented out for future reference. We know hardcoding stuff sucks, but the workarounds presented are not too portable.

wojciesze · 2020-06-16T15:28:52Z

For those who are not happy with dirty tricks in shell script: get_thumbprint.sh (e.g: tac twice).
Here is a working python script: (tested with python3)

import socket
from OpenSSL import SSL
import certifi
import sys, json

hostname = f'oidc.eks.{sys.argv[1]}.amazonaws.com'
port = 443


context = SSL.Context(method=SSL.TLSv1_METHOD)
context.load_verify_locations(cafile=certifi.where())

conn = SSL.Connection(context, socket=socket.socket(socket.AF_INET, socket.SOCK_STREAM))
conn.settimeout(5)
conn.connect((hostname, port))
conn.setblocking(1)
conn.do_handshake()
conn.set_tlsext_host_name(hostname.encode())

thumbprint = conn.get_peer_cert_chain()[-1].digest("sha1")
obj = {"thumbprint": thumbprint.decode("utf-8").replace(":", "").lower() }
print(json.dumps(obj))
conn.close()

For alpine docker images it enough to install:

apk add py3-openssl
pip3 install certifi

For python:3 docker image:

pip3 install pyopenssl certifi

guitarrapc · 2020-06-23T11:03:01Z

#10104 (comment) used to be show longer certificate, which expire 2034.

endpoint: oidc.eks.ap-northeast-1.amazonaws.com

{"thumbprint": "9e99a48a9960b14926bb7f3b02e22da2b0ab7280"}

However I've ran same script and it shows shorter certificate, it expires on Sep 2020. (And is match to certificate show on browser.)

endpoint: oidc.eks.ap-northeast-1.amazonaws.com

{"thumbprint": "b715dc079832da5fc1d4706515be48be79a1c871"}

Certificate Chain

, CN=oidc.eks.ap-northeast-1.amazonaws.com, B715DC079832DA5FC1D4706515BE48BE79A1C871, 08/09/2020 21:00:00
, CN=Amazon, OU=Server CA 1B, O=Amazon, C=US, 917E732D330F9A12404F73D8BEA36948B929DFFC, 19/10/2025 09:00:00
, CN=Amazon Root CA 1, O=Amazon, C=US, 06B25927C42A721631C1EFD9431E648FA62E1E39, 31/12/2037 10:00:00
, CN=Starfield Services Root Certificate Authority - G2, O="Starfield Technologies, Inc.", L=Scottsdale, S=Arizona, C=US, 9E99A48A9960B14926BB7F3B02E22DA2B0AB7280, 29/06/2034 02:39:16
Starfield Class 2 Certification Authority, OU=Starfield Class 2 Certification Authority, O="Starfield Technologies, Inc.", C=US, AD7E1C28B064EF8F6003402014C3D0E3370EB58A, 30/06/2034 02:39:16

marcincuber · 2020-06-23T11:31:45Z

@guitarrapc this is a correct value for your region. B715DC079832DA5FC1D4706515BE48BE79A1C871. The initial value is for a different region.
See previous comment #10104 (comment)

I feel like this is an issue that can now be closed. There are numerous solutions suggested that will solve problems with OIDC provider.

nickatsegment · 2020-07-14T02:08:18Z

Oof, I just created a new cluster today and it has a new cert for us-west-2:

$ region_name=us-west-2; echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0f:a2:dc:f6:0e:ff:f7:41:2f:76:c8:3c:e2:71:0d:6f
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Amazon, OU=Server CA 1B, CN=Amazon
        Validity
            Not Before: Jul 10 00:00:00 2020 GMT
            Not After : Aug 10 12:00:00 2021 GMT

$ region_name=us-west-2; echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print $2}'

7FF5BDFA36CD78827820278B4DABB5E1E97C1476

tyrken · 2020-07-14T10:08:26Z

@nickatsegment I'm thinking this is as you've made the same mistake I made and are using the "leaf" server cert which gets updated owing to it's short expiry. Instead use the root Certificate Authority cert which expires in 2034 or similar.

Switch to use the commands above which have -showcerts in the openssl command line to show all certs in the chain, but beware most force you to use a line reordering utility to pick out the right cert, which changes depending on the OS you run: tac (Linux) or tail -r OS/X.

justcompile · 2020-07-14T10:52:25Z

Taken from a number of sources but with explicit help from this article https://medium.com/@michael.kandelaars/did-your-eks-iam-service-account-roles-break-today-2ea50c869aee, this thread & of course Stackoverflow for bash-fu, I've got this script which grabs the root ca thumbprint.

Not the prettiest, but function over form right?

#!/usr/bin/env bash

echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | awk '/-----BEGIN/{f="cert."(n++)} f{print>f} /-----END/{f=""}'
 
certificates=()

for c in cert.*; do
   certificates+=($(openssl x509 <$c -noout -fingerprint))
done
rm cert.* 

thumbprint=$(echo ${certificates[${#certificates[@]}-1]} | sed 's/://g' | awk -F= '{print tolower($2)}')
thumbprint_json="{\"thumbprint\": \"${thumbprint}\"}"
echo $thumbprint_json

./this_script.sh <aws-region-code>

nickatsegment · 2020-07-14T20:10:31Z

@tyrken ah yep, you're right. Thanks for the pointer.

Here's hoping that TLS provider PR lands sometime in the next century so we can all dispense with the janky awk scripts

olfway · 2020-07-17T06:33:02Z

My external for this

external/thumbprint

#!/bin/bash

set -euo pipefail

HOST="oidc.eks.$1.amazonaws.com"

# https://github.com/terraform-providers/terraform-provider-aws/issues/10104

echo | openssl s_client -servername "$HOST" -showcerts -connect "$HOST:443" 2>&- \
  | sed -n '/-----BEGIN CERTIFICATE-----/h;//!H;$!d;x;s/\(.*-----END CERTIFICATE-----\).*/\1/p' \
  | openssl x509 -fingerprint -sha1 -noout \
  | tr '[:upper:]' '[:lower:]' \
  | sed 's/://g; s/.*=\(.*\)/{"thumbprint": "\1"}/'

Use it like this:

data "external" "thumbprint" {
  program = [
    "/bin/sh",
    "${path.module}/external/thumbprint",
    data.aws_region.current.name,
  ]
}

resource "aws_iam_openid_connect_provider" "this" {
  url = aws_eks_cluster.this.identity[0]["oidc"][0]["issuer"]
  client_id_list = [
    "sts.amazonaws.com",
  ]
  thumbprint_list = [
    data.external.thumbprint.result.thumbprint,
  ]
}

For eu-central-1 region it returns fingerprint for this cert:

Issuer: C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
Validity
    Not Before: Sep  2 00:00:00 2009 GMT
    Not After : Jun 28 17:39:16 2034 GMT
Subject: C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2

ghost · 2020-07-23T17:10:02Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

marcincuber added the enhancement Requests to existing resources that expand the functionality or scope. label Sep 13, 2019

ghost added the service/iam Issues and PRs that pertain to the iam service. label Sep 13, 2019

github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Sep 13, 2019

wdullaer mentioned this issue Sep 24, 2019

Retrieve server certificate thumbprint for embedded eks OIDC provider #10217

Closed

bflad removed the needs-triage Waiting for first response or review from a maintainer. label Sep 24, 2019

alaa mentioned this issue Dec 12, 2019

Add IRSA support terraform-aws-modules/terraform-aws-eks#632

Merged

3 tasks

dannyrandall mentioned this issue Feb 5, 2020

Dependency on KIAM qvest-digital/terraform-kubernetes-alb-ingress-controller#3

Closed

rabidscorpio mentioned this issue May 9, 2020

Add a data source for reading attributes from a servers certificate chain hashicorp/terraform-provider-tls#62

Merged

marcincuber closed this as completed Jun 23, 2020

ghost mentioned this issue Jul 7, 2020

Terraform wants to destroy and re-create on every run lablabs/terraform-aws-eks-alb-ingress#3

Closed

ghost locked and limited conversation to collaborators Jul 23, 2020

Retrieve OIDC provider thumbprint not populating which is required for EKS service accounts #10104

Retrieve OIDC provider thumbprint not populating which is required for EKS service accounts #10104

Comments

marcincuber commented Sep 13, 2019

Description

New or Affected Resource(s)

References

gkrizek commented Sep 18, 2019 • edited

gkrizek commented Sep 19, 2019

zzh8829 commented Sep 24, 2019 • edited

marcincuber commented Sep 24, 2019

zzh8829 commented Sep 24, 2019

dogzzdogzz commented Sep 25, 2019 • edited

matteosilv commented Sep 30, 2019

jmtsi commented Sep 30, 2019 • edited

matteosilv commented Oct 1, 2019

chiefy commented Oct 11, 2019

willthames commented Oct 23, 2019

chiefy commented Oct 23, 2019

dayglojesus commented Oct 30, 2019 • edited

Grejeru commented Oct 31, 2019

dayglojesus commented Oct 31, 2019

rastakajakwanna commented Nov 7, 2019 • edited

willthames commented Nov 7, 2019

mzupan commented Dec 13, 2019

jujugrrr commented Jan 3, 2020

marcincuber commented Jan 3, 2020

jujugrrr commented Jan 3, 2020

cippaciong commented Jan 16, 2020

s0enke commented Jan 23, 2020

mzupan commented Jan 23, 2020

christian-marie commented Mar 3, 2020

groodt commented Mar 5, 2020

scardena commented May 21, 2020 • edited

Grejeru commented May 22, 2020

romaryoricardo commented May 23, 2020 • edited

scardena commented May 25, 2020 • edited

wojciesze commented Jun 16, 2020 • edited

guitarrapc commented Jun 23, 2020 • edited

marcincuber commented Jun 23, 2020 • edited

nickatsegment commented Jul 14, 2020

tyrken commented Jul 14, 2020

justcompile commented Jul 14, 2020 • edited

nickatsegment commented Jul 14, 2020

olfway commented Jul 17, 2020

ghost commented Jul 23, 2020

gkrizek commented Sep 18, 2019 •

edited

zzh8829 commented Sep 24, 2019 •

edited

dogzzdogzz commented Sep 25, 2019 •

edited

jmtsi commented Sep 30, 2019 •

edited

dayglojesus commented Oct 30, 2019 •

edited

rastakajakwanna commented Nov 7, 2019 •

edited

scardena commented May 21, 2020 •

edited

romaryoricardo commented May 23, 2020 •

edited

scardena commented May 25, 2020 •

edited

wojciesze commented Jun 16, 2020 •

edited

guitarrapc commented Jun 23, 2020 •

edited

marcincuber commented Jun 23, 2020 •

edited

justcompile commented Jul 14, 2020 •

edited