Deploying a Production Grade Airflow 2.3.x on AWS-EKS

Dependencies:

• linux
• kubectl
• eksctl
• minikube
• helm
• jinja2
• git
• awscli

Sections

• Section1 - Getting Started
• Section2 - Local Airflow Deployment (Minikube)
• Section3 - Creating Helper Chart using HELM
• Section4 - Creating AWS Resources (EKS, RDS, SSM)
• Section5 - Exposing the UI and Persisting Logs
• Section6 - Cluster Auto Scaler
• Section7 - CI/CD Pipeline
• Section8 - Enabling SSL (Optional)

Cleanup

Do remember to clean up the resources to avoid being billed for when not in use.

Following items are a reminder to make sure you delete them.

Delete EKS

- eksctl delete cluster --name airflow

Delete RDS

- Databases

Delete EC2 resources

- Volumes 
- Load Balancers
- Instances
- Security Group

Delete EFS

- File Systems

Delete SystemsManager

- Parameter Store
    - Parameters

Delete ACM

- Certificates

Delete EKS CloudFormation Stack

- Stacks

Delete CodePipeline

- Pipeline

Delete CodeBuild

- Build Project
- Connections

Delete IAM

- Roles
- Policies

Delete VPC