Support for AAD Workload Identity

[sugibuchi]

Which version of the AzCopy was used?

10.22.2

Which platform are you using? (ex: Windows, Mac, Linux)

• Linux (Debian 12.4)
• AKS Kubernetes cluster enabling AAD Workload Identity
• Run azcopy command in a pod using AAD Workload Identity

What command did you run?

Example 1:

azcopy login --identity

Example 2:

export AZCOPY_AUTO_LOGIN_TYPE=MSI
azcopy list https://*****.blob.core.windows.net/path/to/...

In both cases, azcopy command got stuck without any response.

What problem was encountered?

Authorization by using AAD Workload Identity does not work.

How can we reproduce the problem in the simplest way?

There is no simple way, but...

• Deploy an AKS Kubernetes cluster enabling AAD Workload Identity
• Configure a token federation on a K8S service account
• Deploy a pod with AAD workload Identity
• Run the above command

Have you found a mitigation/solution?

A workaround using AzureCLI login was suggested by #2303 (comment). However, this workaround is problematic in Kubernetes environments since this approach requires an installation of AzureCLI in Docker images, which leads to a significant increase in image sizes and the slowness of pod start.

We prefer a native support of AAD Workload Identity in azcopy which supports the following two methods for authorization:

Option 1: run azcopy login

azcopy login --identity

Option 2: Set env variable AZCOPY_AUTO_LOGIN_TYPE

SET AZCOPY_AUTO_LOGIN_TYPE=MSI

We don't need to explicitly set additional parameters (tenant ID etc.) because the AAD Workload Identity webhook automatically injects all required parameters as the following env variables:

https://azure.github.io/azure-workload-identity/docs/quick-start.html#7-deploy-workload

Environment variable	Description
AZURE_AUTHORITY_HOST	The Azure Active Directory (AAD) endpoint.
AZURE_CLIENT_ID	The client ID of the AAD application or user-assigned managed identity.
AZURE_TENANT_ID	The tenant ID of the registered AAD application or user-assigned managed identity.
AZURE_FEDERATED_TOKEN_FILE	The path of the projected service account token file.

WorkloadIdentityCredential provided by Azure SDK for golang automatically uses values of these env variables.

• https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/azidentity/workload_identity.go#L64-L80

[vibhansa-msft]

Sorry for reverting back late on this. Are you looking for workload identity based login using "azcopy login" command here?
If so, that's something not supported. You can use AzCli instead to login and then azcopy command which can use the token retreived by cli. You just need to point "AZCOPY_AUTO_LOGIN_TYPE" env variable to instruct azcopy to reuse the token.

[sugibuchi]

Are you looking for workload identity based login using "azcopy login" command here?

No. We are looking for native support for AAD workload identity based login in azcopy. Not a workaround for using AAD workload identity based login.

You can use AzCli instead to login

As we and other people have explained in #2112 etc., this workaround using Azure CLI is very problematic in Kubernetes environments, where AAD workload identity is used. We need to add Azure CLI to pod images if we use this workaround in a K8S environment. But Azure CLI is a massive component comprising 50K+ files totalling over 1.2GB. This addition of Azure CLI will significantly increase pod image sizes, leading to the pod start's slowness.

It is technically possible to transparently add AAD workload identity support along with the existing managed identity support. Azure SDK for go supports the AAD workload identity.

• https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/azidentity/workload_identity.go

And NewDefaultAzureCredential in the SDK demonstrates a fallback logic for supporting both AAD workload identity and the managed identity.

• https://github.com/Azure/azure-sdk-for-go/blob/main/sdk/azidentity/default_azure_credential.go#L88-L111

We can find a similar discussion in Azure/azure-cli#26858. Azure CLI already supports authentication using token federations. But Azure/azure-cli#26858 is asking for AAD workload identity support, which internally uses token federations but can be simpler because the AAD workload identity webhook injects all required parameters into execution contexts as environment variables.

[vibhansa-msft]

So what youare suggesting here is in K8s environment adding "az cli" is not feasible due to its size and if azcopy can natively support the workload identity its easier for customer. If I understood correctly the suggested code takes a file as input to read token from and you are ok with that kind of solution. In k8s for this to work you will have to make this file accessible to all pods using azcopy. Is that correct understanding about this ask here?

[pranavmalik-msft]

We are currently assessing the encountered issue and the corresponding feature request. Our aim is to explore more effective solutions to resolve the problem and better meet the requested features.

[sugibuchi]

@vibhansa-msft

In k8s for this to work you will have to make this file accessible to all pods using azcopy.

This is correct. More precisely, Microsoft AAD Workload Identity webhook is responsible for injecting Kubernetes service account tokens into pods as files.

The webhook allows pods to use a service account token projected to a well-known volume path to exchange for an Azure AD access token by leveraging the above properties with the Azure Identity SDKs or the Microsoft Authentication Library (MSAL).

https://azure.github.io/azure-workload-identity/docs/installation/mutating-admission-webhook.html

If I understood correctly the suggested code takes a file as input to read token from and you are ok with that kind of solution.

We are OK for this solution as the AAD Workload Identity is Microsoft's official solution.

Let me precise what "token" means. What AAD Workload Identity webhook injects are not AAD access tokens. The webhook injects Kubernetes service account tokens. The injected service account tokens are not directly used to access Azure resources. But MSAL etc. can use the injected service account tokens to exchange for AAD access tokens.

https://azure.github.io/azure-workload-identity/docs/concepts.html

WorkloadIdentityCredential in Azure SDK for go implements this token exchange logic (read a Kubernetes service account token from a file injected by AAD workload identity webhook -> send the service account token to AAD to get AAD access token).

[vsmalladi]

Any progress on this?

[Kaloszer]

Just hit this, it's a roadblock for me in a deployment process.

[vibhansa-msft]

As of now AzCopy does not support workload identity natively. You have to rely on "azcli" to login and use azcopy with cli based auth mode. We have this item in our backlog but it not on priority right now, due to our other deliverables. We can update you on this in few weeks after re-assessment.

[dfederm]

I also just hit this. We're now no longer allowed to use Service Connections in AzDO with Service Principals so must use Workload Identities. This makes the AzureFileCopy@5 completely unusable in pipelines.

[dfederm]

I was able to work around this by ditching AzCopy in favor of the Azure CLI command "az storage blob upload-batch"

[Kaloszer]

I was able to work around this by ditching AzCopy in favor of the Azure CLI command "az storage blob upload-batch"

This still utilizes AzCopy on the backend I think 🗡️

[grady-bluedot]

trying to leverage Workload identity federation in our service connections in ADO has broken any jobs that utilize the azcopy task, had to revert back all our Service Connections from the from the recommended setup

[gapra-msft]

Hi all, Workload Identity support is currently being tracked for our upcoming May release.

[geekzter]

The new AzureFileCopy@6 task supports workload identity federation. The task also uses Azure RBAC to access blob storage instead. This requires the identity of the service connection used to have the appropriate RBAC role e.g. Storage Blob Data Contributor. See Assign an Azure role for access to blob data.

[ffloimair]

Confirmed working. Thank you very much @geekzter !

[DHowett]

@geekzter I just ran a pipeline build with AzureFileCopy@6, which unfortunately exploded:

##[error]Upload to container: '[REDACTED]' in storage account: '[REDACTED]' with blob prefix: '' failed
with error: 'The term 'azcopy' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path is
correct and try again.' For more info please refer to https://aka.ms/azurefilecopyreadme

azcopy was working properly with @5 :/

[geekzter]

@DHowett if this difference occurs on the same agent, you should create an issue in this repo.

[DHowett]

@geekzter Thanks! I root caused and submitted an issue - microsoft/azure-pipelines-tasks#19705

[JamesBurnside]

As of now AzCopy does not support workload identity natively. You have to rely on "azcli" to login and use azcopy with cli based auth mode. We have this item in our backlog but it not on priority right now, due to our other deliverables. We can update you on this in few weeks after re-assessment.

@vibhansa-msft

Do you know if azcopy used in github actions should support this workflow after login via azure/cli? Our azcli OIDC login command succeeds but the azcopy login fails (https://github.com/Azure/communication-ui-library/actions/runs/9065579627/job/24906469834)

Update: what worked for me was to use az storage blob upload --auth-mode login instead of az storage azcopy blob upload or azcopy copy

[stephenhogg-des]

@JamesBurnside if you set $Env:AZCOPY_TENANT_ID it works. Does for us anyway. It should be documented, but isn't.