Using EKS Pod Identity Agent Credentials

[sethfduke]

I see in the release notes that EKSAuth was added to enable the use of credentials from the eks-pod-identity-agent.

Is there any documentation on how exactly this works with the .Net SDK? Has anyone confirmed that it works? I have tried testing it out, but the it always results in the same errors in the SDK.

In the pod requesting credentials I always see:

System.AggregateException: One or more errors occurred. (Unable to retrieve credentials. Message = \"Unable to reach credentials server\".)

And in the agent running in the cluster, I see:

time="2023-12-12T20:18:18Z" level=info msg="handling new request request from 10.16.31.109:46742" client-addr="10.16.31.109:46742" cluster-name=ncp-sandbox-us-east-1-01
time="2023-12-12T20:18:18Z" level=debug msg="validating call to requested target host 169.254.170.23" client-addr="10.16.31.109:46742" cluster-name=ncp-sandbox-us-east-1-01
time="2023-12-12T20:18:18Z" level=trace msg="Interpreting request target host without port" client-addr="10.16.31.109:46742" cluster-name=ncp-sandbox-us-east-1-01 target-host=169.254.170.23
time="2023-12-12T20:18:18Z" level=error msg="Error fetching credentials: Service account token cannot be empty" client-addr="10.16.31.109:46742" cluster-name=ncp-sandbox-us-east-1-01

I have confirmed my pod does have the environment variables and token mounts present as expected, and the mount does indeed contain the service token, but the AWSSDK client's don't seem to be using it or passing it on to the agent for authentication.

Looking for any feedback or pointers.

[dscpinheiro]

@sethfduke Apologies for the bad experience, but the .NET SDK (and by extension AWS Tools for PowerShell) have not been updated to support this new EKS feature yet. We're planning to work on this next year, but I can't give you an ETA since the actual date might change depending on other priorities.

This documentation page has all the SDKs that currently support EKS Pod Identities: https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html

[sethfduke]

Thanks, appreciate the update. I look forward to the SDK offering full support.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.

[juanchovelezpro]

Hi, is there any plan to support this feature in .NET SDK ?

[dscpinheiro]

Update: We released a new version of AWSSDK.Core (3.7.302) yesterday that supports the updated container credential provider used by EKS Pod Identities. You'll need to update the service clients (e.g. AWSSDK.S3) as well, but the SDK should handle the AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE and AWS_CONTAINER_CREDENTIALS_FULL_URI values set by EKS.

I do recommend reviewing these documentation pages, but please let us know if you run into any issues:

• EKS Pod Identity considerations
• How EKS Pod Identity Agent works with a Pod (especially the note at the bottom: If your workloads currently use credentials that are earlier in the chain of credentials, those credentials will continue to be used even if you configure an EKS Pod Identity association for the same workload)

[dscpinheiro]

Have you followed all the setup instructions required? The SDK should handle the environment variables but there are several steps that must be configured in the EKS cluster itself (i.e. are outside of our control):

• Setting up the Amazon EKS Pod Identity Agent
• Configuring a Kubernetes service account to assume an IAM role with EKS Pod Identity
• Configuring Pods to use a Kubernetes service account

If that doesn't help, I do recommend reaching out to AWS Support, they'll be able to help you.

[sethfduke]

I can confirm that I have had no issues utilizing EKS Pod Identities since this update was released. Much appreciated!

[laiminhtrung1997]

Dear @dscpinheiro
I have an issue with using EKS Pod Identity with the external-dns.
After I associated the IAM Role with ServiceAccount, I deployed the external-dns by using helm install immediately.
The issue is the pod external-dns cannot be mounted on the eks-pod-identity-token, so it cannot do some actions to AWS Service Route53.
I think there is a time delay after associating ServiceAccount with the IAM Role, or maybe something else. I have no idea.
So could someone please help me out with this scenario?

[sethfduke]

@laiminhtrung1997, it would appear that support for EKS Pod Identity in External DNS is available, but requires some modifications to charts to get it to work for now, see kubernetes-sigs/external-dns#4188

external-dns is also written in Go, so I'm not sure asking questions in the .Net library repository would make much sense anyways even if it was an SDK issue.

[laiminhtrung1997]

Dear @sethfduke
Just hoping someone can help me with every threat :(
Thanks a lot anyway.