Using Workload Identity

A demo to show how to use Workload Identity to call Google Cloud API. In this demo we will call the Translate API from a GKE application(pod) using Workload Identity.

Pre-requisites

• Google Cloud Account
• With a Service Account with roles:
  • Kubernetes Engine Admin - to create GKE cluster
  • Service Account roles used to create/update/delete Service Account
    • iam.serviceAccounts.actAs
    • iam.serviceAccounts.get
    • iam.serviceAccounts.create
    • iam.serviceAccounts.delete
    • iam.serviceAccounts.update
    • iam.serviceAccounts.get
    • iam.serviceAccounts.getIamPolicy
    • iam.serviceAccounts.setIamPolicy Or simply you can add Service Account Admin and Service Account User roles
  • Compute Network Admin - to create the VPC networks
• Google Cloud SDK
• terraform
• kubectl
• Taskfile

Optional

• kustomize(Optional)
• direnv(Optional)

Download Sources

Clone the sources,

git clone https://github.com/kameshsampath/workload-identity-gke-demo.git && cd "$(basename "$_" .git)"
export DEMO_HOME="$PWD"

Environment Setup

Variables

When working with Google Cloud the following environment variables helps in setting the right Google Cloud context like Service Account Key file, project etc., You can use direnv or set the following variables on your shell,

export GOOGLE_APPLICATION_CREDENTIALS="the google cloud service account key json file to use"
export CLOUDSDK_ACTIVE_CONFIG_NAME="the google cloud cli profile to use"
export GOOGLE_CLOUD_PROJECT="the google cloud project to use"
export KUBECONFIG="$DEMO_HOME/.kube"

(e.g.)

export CLOUDSDK_ACTIVE_CONFIG_NAME=personal
export GOOGLE_APPLICATION_CREDENTIALS=~/.ssh/my-sa-key.json
export GOOGLE_CLOUD_PROJECT=my-awesome-project
export KUBECONFIG="$DEMO_HOME/.kube"

TIP If you are using direnv you can then create file .envrc.local and add the environment variables. They can then be loaded using direnv allow .

You can find more information about gcloud cli configurations at https://cloud.google.com/sdk/docs/configurations.

As you may need to override few terraform variables that you don't want to check in to VCS, add them to a file called .local.tfvars and set the following environment variable to be picked up by terraform runs,

export TFVARS_FILE=.local.tfvars

NOTE: All .local.tfvars file are git ignored by this template.

Check the Inputs section for all possible variables that are configurable.

Inputs

Name	Description	Type	Default	Required
app_ksa	the kubernetes service account that will be used to run the lingua-greeter deployment	string	"lingua-greeter"	no
app_namespace	the kubernetes namespace where the lingua-greeter demo application will be deployed	string	"demo-apps"	no
app_use_workload_identity	Flag to enable/disable application(pod) from using Workload Identity	bool	false	no
cluster_name	the gke cluster name	string	"my-demos"	no
gke_num_nodes	number of gke nodes	number	2	no
kubernetes_version	the kubernetes versions of the GKE clusters	string	"1.24."	no
machine_type	the google cloud machine types for each cluster node	string	"e2-standard-4"	no
project_id	project id	any	n/a	yes
region	the region or zone where the cluster will be created	string	"asia-south1"	no
release_channel	the GKE release channel to use	string	"stable"	no

Example

An example .local.tfvars that will use a Google Cloud project my-awesome-project, create a two node GKE cluster named wi-demo in region asia-south1 with Kubernetes version 1.24. from stable release channel. The machine type of each cluster node will be e2-standard-4. The demo will be deployed in Kubernetes namespace demo-apps, will use lingua-greeter as the Kubernetes Service Account.

app_ksa            = "lingua-greeter"
app_namespace      = "demo-apps"
cluster_name       = "wi-demo"
configure_app_workload_identity = false
gke_num_nodes      = 2
kubernetes_version = "1.24."
machine_type       = "e2-standard-4"
project_id         = "my-awesome-project"
region             = "asia-south1"
release_channel    = "stable"

NOTE: For rest of the section we assume that your tfvars file is called .local.tfvars

Application Overview

As part of the demo, let us deploy a Kubernetes application called lingua-greeter. The application exposes a REST API /:lang , that allows you to translate a text Hello World! into the language :lang using Google Translate client.

NOTE: The :lang is the BCP 47 language code, https://en.wikipedia.org/wiki/IETF_language_tag.

Tasks

The deployment is done using task. You can list all available tasks using the command,

task --list

Create Environment

We will use terraform to create all the Google Cloud resources like GKE, Service Account, Kubernetes manifests etc.,

task init

Create GKE cluster

The terraform apply will create the following Google Cloud resources,

• A Kubernetes cluster on GKE with Workload Identity enabled
• A Google Cloud VPC that will be used with GKE

task create_cluster

Deploy Application

To see Workload Identity in action we will deploy the application in two parts,

• Application(workload) not enabled for Workload Identity
• Application(workload) enabled for Workload Identity

Create the namespace demo-apps to deploy the lingua-greeter application,

kubectl create ns demo-apps

Run the following command to deploy the application,

kubectl apply -n demo-apps -k $DEMO_HOME/app/config

Wait for application to be ready,

kubectl rollout status -n demo-apps deployment/lingua-greeter --timeout=60s

Get the application service LoadBalancer IP,

kubectl get svc -n demo-apps lingua-greeter

TIP: You can also deploy the application using the command task deploy_app

If the EXTERNAL-IP is <pending> then wait for the IP to be assigned. It will take few minutes for the EXTERNAL-IP to be assigned. You can use the following command to wait until External-IP is assigned,

  while [ -z $(kubectl get svc -n demo-apps lingua-greeter -ojsonpath="{.status.loadBalancer.ingress[*].ip}") ]; do sleep .3; done;

Call Service

export SERVICE_IP=$(kubectl get svc -n demo-apps lingua-greeter -ojsonpath="{.status.loadBalancer.ingress[*].ip}")

Call the service to return the translation of Hello World! in Tamil(ta),

curl "http://$SERVICE_IP/ta"

The service should fail with a message,

{"message":"Internal Server Error"}

When you check the logs of the lingua-greeter pod, you should see a message like,

time="2023-01-25T10:26:50Z" level=error msg="googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.\nMore details:\nReason: authError, Message: Invalid Credentials\n"

As it describes you don't have authentication credentials to call the API. All Google Cloud API requires GOOGLE_APPLICATION_CREDENTIALS to allow client to authenticate itself before calling the API. If you check the deployment manifest we dont have one configured.

Configure Application to use Workload Identity

• Create a Service Account(SA) that has permissions to call Google Translation API, in our demo we call that SA as translator
• Add translator SA with role roles/cloudtranslate.user
• Add an IAM binding policy to translator SA, with the role roles/iam.workloadIdentityUser and a member "serviceAccount:$GOOGLE_CLOUD_PROJECT.svc.id.goog[demo-apps/lingua-greeter]" (default workload identity SA)

Edit your .local.tfvars file and update the app_use_workload_identity to be true.Save the .local.tfvars and run the following command to create the SA, role and IAM policy binding resources,

task use_workload_identity

The command ran earlier should also generate an updated lingua-greeter Kubernetes Service Account manifest $DEMO_HOME/k8s/sa.yaml, that is annotated to impersonate the translator Google SA,

apiVersion: v1
kind: ServiceAccount
metadata:
  name: lingua-greeter
  namespace: demo-apps
  annotations:
    iam.gke.io/gcp-service-account: "translator@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com"

Run the following command to update the Kubernetes SA lingua-greeter to use the Google IAM service Account using Workload Identity mechanics,

kubectl apply -n demo-apps -f "$DEMO_HOME/k8s/sa.yaml"

Call the service again, the service should succeed with a response,

{"text":"Hello World!","translation":"வணக்கம் உலகம்!","translationLanguage":"ta"}

NOTE: Sometimes it may take few seconds for the pods to refresh the metadata, in such cases try to call the service after few seconds.

For more information check out Workload Identity.

Outputs

Name	Description
ksa_patch	The Kubernetes Service Account patch
kubeconfig_path	Kubeconfig file
kubernetes_cluster_host	GKE Cluster Host
kubernetes_cluster_name	GKE Cluster Name
project_id	GCloud Project ID
region	GCloud Region
translator_service_account	The Google Service Account 'translator'
zone	GCloud Zone

License

Apache License