17-CICD-CodePipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center> \n",
    "    <h3><b> Adnan Rashid </b></h3>\n",
    "    <nav>\n",
    "    | <a href=\"00-Contents-Setup.ipynb\"> Home Page </a> | \n",
    "    <a href=\"http://bit.ly/cka_notes_original\"> Kubernetes Notes </a> |\n",
    "    <a href=\"https://adnan.study\"> Website </a> |\n",
    "    <a href=\"https://www.instagram.com/adnans_techie_studies/\"> Instagram </a> |\n",
    "    <a href=\"https://www.linkedin.com/in/adnanrashid1/\"> LinkedIn </a> |\n",
    "    </nav>\n",
    "\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 style=\"background:black\"><code style=\"background:black;color:white\"><center>CI/CD with CodePipeline</center></code></h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "<font style=\"color:black\">\n",
    "    <center> Continous Integration (CI) and Continous Delivery (CD) are essential for companies.<br><br> Teams are more productive when they can make discrete changes frequently, release those changes programmatically and deliver updates without disruption.<br><br> In this module, we will build a CI/CD pipeline using AWS CodePipeline. The CI/CD pipeline will deploy a sample Kubernetes service, we will make a change to the GitHub repository and observe the automated delivery of this change to the cluster.  \n",
    "</center>\n",
    "</font>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"https://adnanstudyimages.s3-eu-west-1.amazonaws.com/32.jpg\" width=\"50%\" height=\"50%\" border=5/></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-requisite"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Check your variables are loaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Load in variable store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Load variables into notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%env AWS_PROFILE=$AWS_PROFILE\n",
    "%env AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION\n",
    "%env ACCOUNT_ID=$ACCOUNT_ID"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Check identity "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws sts get-caller-identity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create IAM Role"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In an AWS CodePipeline, we are going to use AWS CodeBuild to deploy a sample Kubernetes service. \n",
    "\n",
    "This requires an AWS IAM role capable of interacting with the EKS cluster. \n",
    "\n",
    "In this step, we are going to create an IAM role and add an inline policy that we will use in the CodeBuild stage to interact with the EKS cluster via kubectl. \n",
    "\n",
    "* Create the role"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash \n",
    "TRUST=\"{ \\\"Version\\\": \\\"2012-10-17\\\", \\\"Statement\\\": [ { \\\"Effect\\\": \\\"Allow\\\", \\\"Principal\\\": { \\\"AWS\\\": \\\"arn:aws:iam::${ACCOUNT_ID}:root\\\" }, \\\"Action\\\": \\\"sts:AssumeRole\\\" } ] }\"\n",
    "echo '{ \"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Action\": \"eks:Describe*\", \"Resource\": \"*\" } ] }' > codepipeline/iam-role-policy\n",
    "\n",
    "aws iam create-role --role-name EksWorkshopCodeBuildKubectlRole --assume-role-policy-document \"$TRUST\" --output text --query 'Role.Arn'\n",
    "\n",
    "aws iam put-role-policy --role-name EksWorkshopCodeBuildKubectlRole --policy-name eks-describe --policy-document file://codepipeline/iam-role-policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Modify AWS-Auth ConfigMap "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the IAM role created, we are going to add the role to the aws-auth ConfigMap for the EKS clsuter. \n",
    "\n",
    "Once the ConfigMap includes this new role, kubectl in the CodeBuild stage of the pipeline will be able to interact with the EKS cluster via the IAM role"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash \n",
    "\n",
    "ROLE=\"    - rolearn: arn:aws:iam::$ACCOUNT_ID:role/EksWorkshopCodeBuildKubectlRole\\n      username: build\\n      groups:\\n        - system:masters\"\n",
    "\n",
    "kubectl get -n kube-system configmap/aws-auth -o yaml | awk \"/mapRoles: \\|/{print;print \\\"$ROLE\\\";next}1\" > codepipeline/aws-auth-patch.yml\n",
    "kubectl patch configmap/aws-auth -n kube-system --patch \"$(cat codepipeline/aws-auth-patch.yml)\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to edit the aws-auth ConfigMap manually, you can run the following"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"background:black\">\n",
    "<code style=\"background:black;color:white\">> kubectl edit -n kube-system configmap/aws-auth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to fork the sample Kubernetes service so that we will be able to modify the repository and trigger builds. \n",
    "\n",
    "* Login to Github and fork the sample service to your own account "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "https://github.com/rnzsgh/eks-workshop-sample-api-service-go"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <br><br>Github Access Token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order for CodePipeline to receive callbacks from GitHub, we need to generate a personal access token. \n",
    "\n",
    "Once created, an access token can be stored in a secure enclave and reused, so this step is only required during the first run or when you need to generate new keys. \n",
    "\n",
    "Open up the <a href=\"https://github.com/settings/tokens/new\"> New personal access page </a> in GitHub and do following\n",
    "\n",
    "* Enter a value for <b> Token Description </b> \n",
    "* Check the <b>repo</b> permission scope and scroll down and click the <b> Generate token </b> button "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"https://adnanstudyimages.s3-eu-west-1.amazonaws.com/github_token_name.png\" width=\"70%\" height=\"70%\" border=5/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Copy the <b> personal access token </b> and save it in a secure plae for the next step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"https://adnanstudyimages.s3-eu-west-1.amazonaws.com/github_copy_access.png\" width=\"70%\" height=\"70%\" border=5/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <br><br> CodePipeline Setup "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to create the AWS CodePipeline using iAWS CloudFormation. \n",
    "\n",
    "CloudFormation is an Infrastructure as Code (IaC) tool which provides a common language for you to describe and provision all the infrastructure resources in your cloud enviornment. CloudFormation allows you to use a simple text file to model and provision, in an automated and secure manner, all the resources needed for your applications across all regions and accounts. \n",
    "\n",
    "Each EKS deployment/service should have its own CodePipeline and be located in an isoalted source repository. \n",
    "\n",
    "You can modify the CloudFormation template provided to meet your system requirements to easily onboard new services to your EKS cluster. For each new service the following steps can be repeated. \n",
    "\n",
    "* Deploy the following stacks by copying to browser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "https://console.aws.amazon.com/cloudformation/home?#/stacks/create/review?stackName=eksws-codepipeline&templateURL=https://s3.amazonaws.com/eksworkshop.com/templates/main/ci-cd-codepipeline.cfn.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* After the console is open, enter your Github username and personal access toekn (created earlier), check the acknowledge box and then click the 'Craete Stack' button located at the bottom of the page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"https://adnanstudyimages.s3-eu-west-1.amazonaws.com/cloudformation_stack.png\" width=\"100%\" height=\"100%\" border=5/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Wait for the status to change from 'CREATE_IN_PROGRESS' to <b> CREATE_COMPLETE </b> before moving on to the next step. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"https://adnanstudyimages.s3-eu-west-1.amazonaws.com/cloudformation_stack_creating.png\" width=\"100%\" height=\"100%\" border=5/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Open <a href=\"https://console.aws.amazon.com/codesuite/codepipeline/pipelines\"> Codepipeline in the management console </a>. You will see a CodePipeline that starts with <b> eks-workshop-codepipeline </b>. \n",
    "\n",
    "Note: Ensure you are in the correct region where it was deployed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"https://adnanstudyimages.s3-eu-west-1.amazonaws.com/codepipeline_landing.png\" width=\"100%\" height=\"100%\" border=5/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Once you are on the detail page for the specific CodePipeline, you can see the status along with the links to the change and build details. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"https://adnanstudyimages.s3-eu-west-1.amazonaws.com/codepipeline_details.png\" width=\"100%\" height=\"100%\" border=5/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* To review that status of the deployment you can run the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe deployment hello-k8s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* For the status of the service, run following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe service hello-k8s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Once the service is built and delivered, we can run the following command to get the Elastic Load Balancer (ELB) endpoint and open it in a browser. \n",
    "\n",
    "If the message is not updated immediately, give Kubernetes some time to deploy the change "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get service hello-k8s -o wide"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Check the URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl a79bbc651d95e4c6e8fdb54b07aa77e5-995409802.eu-west-1.elb.amazonaws.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <br><br> Trigger New Release "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far we have walked through setting up CI/CD for EKS using AWS CodePipeline and now we are going to make a change to the Github Repository so that we can see a new release built and delivered. \n",
    "\n",
    "* Clone the repository"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "https://github.com/arashid290/eks-workshop-sample-api-service-go.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make a change to the <b> main.go </b> file and change the text where it says 'Hello World' \n",
    "\n",
    "* Before "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "func main() {\n",
    "\thttp.HandleFunc(\"/\", func(w http.ResponseWriter, r *http.Request) {\n",
    "\n",
    "\t\tf := fib()\n",
    "\n",
    "\t\tres := &response{Message: \"Hello World\"}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* After "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "func main() {\n",
    "\thttp.HandleFunc(\"/\", func(w http.ResponseWriter, r *http.Request) {\n",
    "\n",
    "\t\tf := fib()\n",
    "\n",
    "\t\tres := &response{Message: \"New Message\"}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Commit and push the changes "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "git commit -am 'change the message' \n",
    "git push "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "After you modify and commit the change in Github, in approximately 1 minute you will see a new build triggered in AWS "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"https://adnanstudyimages.s3-eu-west-1.amazonaws.com/codepipeline_building.png\" width=\"100%\" height=\"100%\" border=5/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Lets check the website once the build is complete and you will notice at the top the message has changed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl a79bbc651d95e4c6e8fdb54b07aa77e5-995409802.eu-west-1.elb.amazonaws.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <br><br> Clean up "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Delete the Kubernetes deployment and service "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl delete deployments hello-k8s\n",
    "!kubectl delete services hello-k8s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Delete the Cloudformation stack created in the console "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws cloudformation delete-stack --stack-name eksws-codepipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Delete the ECR Repository find your repo first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws ecr describe-repositories --output \"text\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws ecr delete-repository --repository-name ecrdockerrepository-27eh2ehcjlka --force"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Delete S3 bucket "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 delete-bucket --bucket-name eksws-codepipeline*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 ls | sort"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Remove contents from bucket "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 rm s3://eksws-codepipeline-codepipelineartifactbucket-1n3alcqr9so8k --recursive "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3api delete-bucket --bucket eksws-codepipeline-codepipelineartifactbucket-1n3alcqr9so8k"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}