Kubernetes, also known as K8s, is an open source system for managing [containerized applications] across multiple hosts. It provides basic mechanisms for deployment, maintenance, and scaling of applications.
In simple terms, Kubernetes helps in managing the lifecycle of containerized applications by providing a wide range of features and tools. These include load balancing, rolling updates, self-healing, horizontal scaling, and desired state management. Kubernetes groups containers together to form logical units, called pods, which can be easily managed and scaled. It also provides a declarative approach to manage the infrastructure, allowing users to define the desired state of the system through configuration files.
Kubernetes has become the industry standard for container orchestration due to its powerful features, extensive ecosystem, and strong community support. It works with various container runtime engines, such as Docker and containerd, and is compatible with many cloud platforms, making it a highly versatile and sought-after solution for managing containerized applications.
This project aims to exemplify the Kubernetes fundamentals with pratical manifests. We deployed a simple Go app, in which is the base to test most part of Kuberentes features. We also deploy MySQL for Stateful examples. For the whole doc, we are using Kind (local Kubernetes clusters using Docker container “nodes”) to quickly create a cluster, validade all concepts and explore the K8s features without need too much boilerplate.
We will separate it per modules, and each topic from modules will have visual examples following the construction timeline (via commits). This means that every new feature will have a commit related to it, this way its better to visualize all the changes and follow the project construction flow
We have a main Makefile with all the step commands to exemplify and automate the process of cluster creation, shortcuts for deploy, stress test and validation concepts executing scripts. You can take a better look here
Kind, which stands for Kubernetes IN Docker, is an open-source tool that enables users to create and manage lightweight Kubernetes clusters using Docker containers. It serves as a valuable resource for testing and development purposes, as it allows for rapid and efficient deployment of Kubernetes clusters on local machines, streamlining the testing and debugging process for developers and DevOps professionals.
A KIND configuration file is a YAML-formatted file that provides a declarative way to define the desired state and structure of a Kubernetes cluster created using the KIND (Kubernetes IN Docker) tool.
Command to create the cluster:
kind create cluster --config kind-config.yaml --name $(NAMESPACE)Related Commit: feat: ✨ Added kind config yaml
In Kubernetes (K8s), a pod is the smallest and most basic unit in the K8s object model. A pod represents a single instance of a running process in a cluster and typically contains one or more containers that work closely together to deliver a specific function or part of an application.
A ReplicaSet in Kubernetes is a higher-level abstraction over pods that ensures a specified number of replicas of a pod are running at any given time. In simple terms, a ReplicaSet ensures that a certain number of identical pod instances are running, and if any of those instances fail or are terminated, the ReplicaSet will create new ones to maintain the desired count.
Related commit: feat: ✨ Replicasets: higher-level abstraction over pods
A Deployment in Kubernetes is a resource that manages the desired state, scaling, and updates of ReplicaSets and their associated pods, ensuring the continuous availability and stability of applications.
Related commit feat: ✨ Deployments: Manages the desired state, scaling, and updates of ReplicaSets
"Service" in the context of Kubernetes (often abbreviated as "k8s" for convenience) refers to an abstraction that represents a set of pods that perform a similar function and provide a consistent endpoint for client applications to access them.
ports:
- name: go-http-app-backend
port: 80
targetPort: 8000Port and targetPort are key concepts when configuring services in Kubernetes. The "port" defines the external port on which the service is exposed and accessible within the cluster. The "targetPort" specifies the port on which the backend pods or containers are listening for incoming traffic. When a connection is made to the service at the specified "port", traffic is forwarded to the corresponding "targetPort" on the matched backend pods.
Related commit: feat: ✨ Services: Port and TargetPort
NodePort is a type of Kubernetes Service that exposes an application to external clients by assigning a static port on each node in the cluster. This allows users to access the service through any node's IP address and the assigned NodePort, making it a convenient way to expose applications running within the cluster to external traffic.
port: 80
targetPort: 8000
protocol: TCP
type: ClusterIP
type: NodePortRelated commit: feat: ✨ Service: NodePort
A load balancer is a networking component that distributes incoming traffic across multiple backend servers, ensuring optimal resource utilization and high availability. In the context of Kubernetes, a LoadBalancer service type is used to expose applications to external clients by automatically provisioning a cloud provider's load balancer, which directs traffic to the appropriate Pods based on their readiness and load balancing algorithms.
port: 80
targetPort: 8000
protocol: TCP
type: NodePort
type: LoadBalancerExample Commit: feat: ✨ Services: LoadBalancer
ClusterIP is a type of Service in Kubernetes that exposes the service on a cluster-internal IP address. This IP address is only accessible within the cluster and can be used by other services or pods in the cluster to access the service. When you create a Service with a ClusterIP, Kubernetes assigns a stable IP address to the service, which allows other services to communicate with it without having to know the IP address of the underlying pods.
port: 80
targetPort: 8000
protocol: TCP
type: NodePort
type: ClusterIPIn Kubernetes deployments, environment variables enable flexible application configuration within containers. They help decouple applications from infrastructure, making them more portable and easily managed through the deployment manifest, without modifying the container image.
Related commit: feat: ✨ Config: Environment Vars
ConfigMaps in Kubernetes are a flexible way to store and manage non-sensitive configuration data for your applications, enabling you to decouple configuration settings from container images and simplifying updates and maintenance.
-
Related commit: feat: ✨ Config: Configmaps
-
Related commit: feat: ✨ Configs: Confimap - envFrom (import all values->keys)
-
Related commit: feat: ✨ Config: Configmaps - File injection with Volumes
Kubernetes Secrets are secure objects used to store sensitive data, such as passwords, API keys, or tokens, within a cluster. They help protect confidential information and reduce the risk of exposing it accidentally. Secrets store data in base64 encoding, ensuring an additional layer of obfuscation. They can be mounted as files, used as environment variables, or accessed by the Kubernetes API, providing a secure and controlled way to share sensitive information with containers.
Related commit: feat: ✨ Configs: Secrets
"Probes" in Kubernetes (often abbreviated as "k8s" for convenience) refer to a mechanism that allows you to determine the health of a container running inside a pod. Kubernetes provides three types of probes:
A liveness probe in Kubernetes is a mechanism to determine the health of a running container by periodically checking its responsiveness. If a container fails the liveness check, Kubernetes will restart it, ensuring that the application remains available and recovers from potential issues automatically. In this example, an HTTP GET request is made to the /healthz endpoint on port 8000 every 5 seconds. If the container fails the liveness check, Kubernetes restarts it after reaching the failure threshold of 1. The probe waits up to 1 second for a response before timing out, and the container is considered healthy after a single successful check. This ensures that the application remains available and automatically recovers from potential issues.
Related commit: feat: ✨ Probes: Liveness
A readiness probe in Kubernetes is used to determine if a container is ready to accept incoming traffic. By periodically checking the container's responsiveness, Kubernetes can prevent traffic from being sent to a container that is not yet fully operational, ensuring a more stable and reliable application experience for users.
-
Related commit: feat: ✨ Probes: Readiness
-
Related commit: feat: ✨ Probes: Readiness + Liveness combined
A startup probe in Kubernetes is a mechanism to determine if a container has successfully started and initialized. It is particularly useful for slow-starting containers that require additional time to become fully operational. Kubernetes uses the startup probe to monitor the container's initialization process, and once the container passes the startup check, it transitions to liveness and readiness checks, ensuring the application has enough time to initialize before serving traffic.
-
Related commit: feat: ✨ Probes: Readiness & Liveness solution: StartupProbe
-
Related commit: refactor: ♻️
The Metrics Server is a crucial Kubernetes component that collects and stores resource usage metrics such as CPU and memory for nodes and pods within a cluster. It enables efficient resource management, autoscaling, and basic monitoring, helping to ensure that applications scale according to performance demands and making it easier to identify issues.
Related commit: feat: ➕ HPA: Installing metrics-server
Resource requests and limits in Kubernetes are essential tools for managing and optimizing the allocation of CPU and memory resources for your containers. Requests define the minimum amount of resources needed for the container to run, while limits set an upper boundary to prevent excessive resource consumption. By carefully configuring requests and limits, you can ensure that your applications run efficiently, maintain high performance, and coexist harmoniously within your cluster.
Related commit: feat: ✨ HPA: Resources: Requests and Limits
Horizontal Pod Autoscaling (HPA) is a Kubernetes feature that automatically adjusts the number of running pods in a deployment or replica set based on real-time metrics such as CPU utilization or custom metrics. This ensures that your application can scale out to handle increased load and scale in when demand decreases, optimizing resource usage and performance. (Command to perform CPU tests: kubectl run -it fortio --rm --image=fortio/fortio -- load -qps 800 -t 120s -c 70 "http://go-http-app-service/healthz")
-
Related commit: feat: ✨ HPA: Horizontal Pod Autoscaler
-
Related commit: feat: ✨ HPA: Stress test w/ Fortio (Makefile automation)
A Persistent Volume Claim (PVC) is a Kubernetes resource that allows users to request and consume storage from available Persistent Volumes in the cluster, providing an abstraction layer and efficient storage management for containerized applications.
-
Related commit: feat: ✨ Persistent Volume: Claim
-
Related commit: refactor: ♻️ Code Refactor: Names and metrics
A StatefulSet is a Kubernetes resource used to manage stateful applications, ensuring a stable network identity and stable storage for each of its replicas. When used with a headless service, the StatefulSet allows direct access to each Pod via a unique DNS name, facilitating communication between replicas and enabling service discovery without load balancing or a cluster IP. This combination is commonly employed for applications like databases, which require consistent network identities and persistent storage for data.
Related commit: feat: ✨ Statefulset: MySQL example (Headless)
The NGINX Ingress Controller is a popular implementation of the Kubernetes Ingress concept, which provides external access to cluster services by routing incoming HTTP/HTTPS traffic to the appropriate backend services. By leveraging the power of the NGINX reverse proxy and load balancer, the Ingress Controller efficiently manages traffic and allows for custom routing rules and annotations. This enables developers to effectively expose and secure their applications within a Kubernetes cluster, while adhering to best practices for scalability and high availability
-
Related commit: feat: ➕ Ingress: Install Nginx
-
Related commit: feat: ✨ Ingress: Nginx ingress
Cert Manager is a Kubernetes-native certificate management solution that automates the issuance, renewal, and management of SSL/TLS certificates for your applications. By integrating with various certificate authorities (CAs) such as Let's Encrypt or self-signed certificates, Cert Manager ensures secure communication between clients and services within a Kubernetes cluster. This simplifies certificate lifecycle management and helps maintain security best practices, allowing developers to focus on building and deploying their applications with confidence.
Related commit: feat: ✨ Cert-Manager: Install
A ClusterIssuer is a Kubernetes resource used by Cert-Manager to define a Certificate Authority (CA) for issuing SSL/TLS certificates across the entire cluster. It is a cluster-wide variant of the namespaced Issuer resource. ClusterIssuers streamline certificate management by allowing a single resource to provide certificates to applications in different namespaces, enabling consistent and centralized certificate lifecycle management. By integrating with various CAs, such as Let's Encrypt or by utilizing self-signed certificates, ClusterIssuers ensure that applications within the Kubernetes cluster can secure their communications with clients, promoting best practices for security and reliability.
-
Related commit: feat: ✨ Cert-Manager: Adding a ClusterIssuer
-
Related commit: refactor: ♻️ Refactoring ingress name
The TLS settings in this file ensure secure communication (HTTPS) between clients and the cluster by leveraging Let's Encrypt as the certificate issuer. We define the "secretName" as the generated secret from ClusterIssuer (cert-manager). We are using an example DNS, this means that the certificate will be issued, but not generated (since the example DNS is not configured to point to our k8s external ip)
Related commit: feat: ✨ TLS: Adapt Ingress to TLS configs
In Kubernetes, namespaces provide a logical separation of resources, allowing you to manage and organize your workloads more effectively while maintaining a clear boundary between different projects or teams.
-
Related commit: feat: ✨ Creating the cluster's namespaces
-
Related commit: feat: ✨ Namespace: Defining namespace for each manifest
Utilize context on Kubernetes to seamlessly switch between multiple clusters or namespaces, enabling increased productivity and streamlined management of diverse environments. For this example, i create a Makefile automation to create the config-contexts based on namespaces (server and database for our example) and three others in order to use database, server ou default context.
Related commit: feat: ✨ Namespace: Using K8s Conext based in Namespace
In Kubernetes, a service account is an identity that is used by pods and other resources in a cluster to authenticate with the Kubernetes API server and to access other cluster resources.
In Kubernetes, a Role is a set of permissions that defines what actions can be performed on specific resources in a namespace. A Role can be associated with a ServiceAccount, which allows Pods that use that ServiceAccount to perform the actions defined by the Role. A Role consists of a set of rules that specify which API groups, resources, and verbs the Role can access. For example, a Role might allow read-only access to Pods and ConfigMaps, but not allow write access to Secrets. Roles are namespace-specific, meaning that they only apply to resources within a specific namespace. This allows you to control which resources can be accessed by Pods associated with a specific ServiceAccount within a given namespace.
Related commit: feat: ✨ ServiceAccount: Creating Auth Role (list services on server ns)
RoleBinding is a way to bind a Role to a user, group, or ServiceAccount within a namespace. A RoleBinding grants the permissions defined in a Role to the entity associated with the RoleBinding. A RoleBinding consists of two parts: the Role that defines the permissions, and the subject that defines the entity to which the permissions are granted. The subject can be a user, group, or ServiceAccount.
Related commit: feat: ✨ ServiceAccount: Added Role Binding
Now that we identified the serviceAccountName into the Deployment, that means that all the pods from that deployment will have the server-service-account permissions based on namespace. For now, if we run "make test-serviceaccount-pod-list", we will see the tests working and returning the service list from our k8s API (allowed by the svc account).
Related commit: feat: ✨ ServiceAccount: Defining the Svc Account on Deployment
Here we have a simple script injection into our Go app in order to do a CURL requisition to local K8S Api from within the Pod. This will be important to validate that if we can access it when ServiceAccount its added.
-
Related commit: test: ⚗️ ServiceAccount: Inject script to test k8s API access (curl)
-
Related commit: feat: 🔨 Makefile step to execute the injected script within server pod automatically. Use
make test-serviceaccount-pod-listto test
Here we define exactly how the command should behave and what conclusions we can get. The first part of the script will be testing if the pods with namespace "server" can access the K8S API in order to list Pods (expected: not allowed). The second part of it will check if the "server" pods can access the Service listing from k8s API (expected: Allowed)
Related commit: feat: ♻️ ServiceAccount: Refactor the SvcAcc test script
The first part of the script will be testing if the pods with namespace "server" can access the K8S API in order to list Pods (expected: not allowed). The second part of it will check if the "server" pods can access the Service listing from k8s API (expected: Allowed)
Make Command:
make test-serviceaccount-pod-listExpected result:
=======================================================
< ====== STARTING DENIED CALL (List Pods) ====== >
=======================================================
-> ACTION 1: PODS listing from Kubernetes Api (NOT Allowed): -----------------------------------------
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "pods is forbidden: User \"system:serviceaccount:server:server-service-account\" cannot list resource \"pods\" in API group \"\" in the namespace \"server\"",
"reason": "Forbidden",
"details": {
"kind": "pods"
},
"code": 403
}-X END ----------------------------------------------------------------------------------
=======================================================
< ====== STARTING ALLOWED CALL (List Services) ====== >
=======================================================
-> ACTION 2: SERVICES listing from Kubernetes Api (Allowed): -----------------------------------------
{
"kind": "ServiceList",
"apiVersion": "v1",
"metadata": {
"resourceVersion": "1876"
},
"items": [
{
"metadata": {
"name": "go-http-app-service",
"namespace": "server",
"uid": "c20d0531-7ab0-4354-b6f5-8f2bc9dff8d3",
"resourceVersion": "1071",
"creationTimestamp": "2023-03-22T15:27:20Z",
"labels": {
"app": "go-http-app",
"tier": "backend"
},
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"go-http-app\",\"tier\":\"backend\"},\"name\":\"go-http-app-service\",\"namespace\":\"server\"},\"spec\":{\"ports\":[{\"name\":\"go-http-app-backend\",\"port\":80,\"protocol\":\"TCP\",\"targetPort\":8000}],\"selector\":{\"tier\":\"backend\"},\"type\":\"ClusterIP\"}}\n"
},
"managedFields": [
{
"manager": "kubectl-client-side-apply",
"operation": "Update",
"apiVersion": "v1",
"time": "2023-03-22T15:27:20Z",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:annotations": {
".": {},
"f:kubectl.kubernetes.io/last-applied-configuration": {}
},
"f:labels": {
".": {},
"f:app": {},
"f:tier": {}
}
},
"f:spec": {
"f:internalTrafficPolicy": {},
"f:ports": {
".": {},
"k:{\"port\":80,\"protocol\":\"TCP\"}": {
".": {},
"f:name": {},
"f:port": {},
"f:protocol": {},
"f:targetPort": {}
}
},
"f:selector": {},
"f:sessionAffinity": {},
"f:type": {}
}
}
}
]
},
"spec": {
"ports": [
{
"name": "go-http-app-backend",
"protocol": "TCP",
"port": 80,
"targetPort": 8000
}
],
"selector": {
"tier": "backend"
},
"clusterIP": "10.96.250.10",
"clusterIPs": [
"10.96.250.10"
],
"type": "ClusterIP",
"sessionAffinity": "None",
"ipFamilies": [
"IPv4"
],
"ipFamilyPolicy": "SingleStack",
"internalTrafficPolicy": "Cluster"
},
"status": {
"loadBalancer": {}
}
}
]
}-X END ----------------------------------------------------------------------------------