Amazon API Gateway Mutating Webhook For K8S

This demo project is intended to illustrate how to use Amazon API Gateway and AWS Lambda to set up an HTTP service, then been integrated with Kubernetes as admission webhooks to receive admission requests and mutate or validate Kubernetes resources dynamically. Particularly this project will setup a mutating webhook to modify the docker image path in K8S Pod after the deployment been submitted to K8S API server and before it's been persisted in etcd.

Use cases

• The same k8s cluster need to be deployed and use different docker registry, for example k8s cluster deployed in AWS Oregon and Singapore region, and use ECR registry in local region.
• Due to firewall or security restriction, public registry cannot be accessed and need to modify the image path to access other mirrored repositories.

How to deploy

Prerequisites

• Make sure that the Kubernetes cluster is at least as new as v1.9.
• Make sure that MutatingAdmissionWebhook admission controllers are enabled.
• Make sure that the admissionregistration.k8s.io/v1beta1 API is enabled.
• If you are using AWS China regions, make sure your AWS account has been whitelisted for API Gateway access, you may contact AWS support for the procedure of the whitelisting.

Amazon EKS has been enabled MutatingAdmissionWebhook.

Steps to set up admission webhook:

1. Setup webhook with API GW, there are 2 options to build the API GW:

   OPTION #1 - Install from SAR(Serverless App Repository)

   This is the recommended approach, deploy it from SAR console:

   Region	Click and Deploy
   ap-northeast-1
   ap-east-1
   ap-northeast-2
   ap-northeast-3
   ap-south-1
   ap-southeast-1
   ap-southeast-2
   ca-central-1
   eu-central-1
   eu-north-1
   eu-west-1
   eu-west-2
   eu-west-3
   me-south-1
   sa-east-1
   us-east-1
   us-east-2
   us-west-1
   us-west-2
   cn-north-1
   cn-northwest-1

   OPTION #2 - Build from scratch

   Firstly please install AWS CLI and SAM CLI by following AWS SAM documentation

   1). Check out this repository

   $ git clone https://github.com/aws-samples/amazon-api-gateway-mutating-webhook-for-k8.git
   $ cd amazon-api-gateway-mutating-webhook-for-k8

   2). Build the SAM package, replace my_s3_bucket with your S3 bucket name at first.

   $ export S3_BUCKET=my_s3_bucket
   $ sam package -t sam-template.yaml --s3-bucket ${S3_BUCKET} --output-template-file packaged.yaml

   3). Deploy SAM package

   $ sam deploy -t packaged.yaml --stack-name amazon-api-gateway-mutating-webhook-for-k8 --capabilities CAPABILITY_IAM

2. Go to the cloudformation stack console, find the stack which name is serverlessrepo-amazon-api-gateway-mutating-webhook-for-k8s if you are are using SAR deployment in previous step, or find the stack with name amazon-api-gateway-mutating-webhook-for-k8, get APIGatewayURL from the outputs of this stack.

3. Create k8s MutatingWebhookConfiguration resource

   • Modify mutating-webhook.yaml，replace with the value of APIGatewayURL
   • Create K8S resource:

     $ kubectl apply -f mutating-webhook.yaml

4. Deploy sample k8s deployment

   $ kubectl apply -f ./nginx-gcr.yaml

5. Check the image path

   $ kubectl get pod nginx-gcr-deployment-784bf76d96-hjmv4 -o=jsonpath='{.spec.containers[0].image}'
   asia.gcr.io/nginx

   you may noticed the image path has been changed from "gcr.io/nginx" to "asia.gcr.io/nginx"

How it works

The most famous use case of k8s mutating webhook is the istio sidecar injector, you may also reference K8S documentation or 3rd-party blog to get understand how k8s admission works.

This project will setup the API Gateway with Lambda to receive the admission request from k8s API server, you may get k8s Pod spec from "body" object of request payload json object:

{
    "kind": "AdmissionReview",
    "request": {
        "kind": {
            "kind": "Pod",
            "version": "v1",
            "group": ""
        },
        "resource": {
            "resource": "pods",
            "version": "v1",
            "group": ""
        },
        "uid": "b06b6ec2-681d-11e9-a645-06b44ed6a042",
        "object": {
            "status": {},
            "spec": {
                "dnsPolicy": "ClusterFirst",
                "securityContext": {},
                "serviceAccountName": "default",
                "schedulerName": "default-scheduler",
                "serviceAccount": "default",
                "priority": 0,
                "terminationGracePeriodSeconds": 30,
                "restartPolicy": "Always",
                "containers": [
                    {
                        "name": "nginx",
                        "image": "gcr.io/nginx:latest",
                        "imagePullPolicy": "Always",
                        "ports": [
                            {
                                "protocol": "TCP",
                                "containerPort": 80
                            }
                        ],
                        "resources": {}
                    }
                ]
            },
            "metadata": {
            }
        },
        "namespace": "admission-test",
        "userInfo": {
            "username": "system:unsecured",
            "groups": [
                "system:masters",
                "system:authenticated"
            ]
        },
        "oldObject": null,
        "dryRun": false,
        "operation": "CREATE"
    },
    "apiVersion": "admission.k8s.io/v1beta1"
}

then the response of webhook(API Gateway with Lambda) is like this:

{
    "body": {
    "kind": "AdmissionReview",
    "apiVersion": "admission.k8s.io/v1",
    "response": {
        "uid": "request.uid",
        "allowed": "True",
        "patch": "patch_base64",
        "patchType": "JSONPatch"
    },
    "headers": {
      "Content-Type": "application/json"
    },
    "statusCode": 200
}

where the patch_base64 is base64 encoded of JSON Patch, for example:

[
    {
        "op": "replace",
        "path": "/spec/containers/0/image",
        "value": "xxxx.dkr.ecr.us-west-2.amazonaws.com/nginx:latest"
    }
]

JSON patch will be applied to the Pod spec and then persisted into etcd.

License Summary

This sample code is made available under the MIT-0 license. See the LICENSE file.