This project offers a reference for deploying a Kubernetes cluster. Its ansible playbooks allow to provision a deployment on Bare-metal or Virtual Machines.
The POD Descriptor File (PDF) is a YAML file which defines the setup used by KRD. The default PDF file can be overwritten to try different scenarios. There are different setups located in the samples folder which can be used.
| Name | Version |
|---|---|
| Ubuntu | 18.04/20.04/22.04 |
| CentOS | 7/8 |
| openSUSE | Leap |
| Fedora | 37/38 |
| Name | Description | Source | Status |
|---|---|---|---|
| Kubernetes | Base Kubernetes deployment | kubespray | Done |
| Virtlet | Allows to run VMs | configure-virtlet.yml | Tested |
| Kubevirt | Allows to run VMs | Tested | |
| Virtink | Allows to run VMs | Tested | |
| Multus | Provides Multiple Network support in a pod | Tested | |
| NFD | Node feature discovery | nfd role | Tested |
| Istio | Service Mesh platform | Tested | |
| PMEM | Persistent Memory CSI | pmem role | Implemented |
| QAT | QuickAssist Technology Plugin | qat_plugin role | Tested |
| SR-IOV | Single Root Input/Output Virtualization Plugin | sriov_plugin role | Implemented |
| Knative | Platform for managing serverless workloads | Tested | |
| Kong | Ingress Controller | Tested | |
| HAProxy | Ingress Controller | Tested | |
| MetalLB | Provides a network load-balancer implementation | Tested | |
| Falco | Deploys a Falco services | Tested | |
| Rook | Storage Operator | Tested | |
| OPA Gatekeeper | Policy Controller | Tested | |
| Kyverno | Kubernetes Native Policy Management | Tested | |
| Kubewarden | Kubernetes Native Policy Management | Tested | |
| kube-monkey | Netflix's Chaos Monkey for Kubernetes clusters | Implemented | |
| Nephio | Management platform to on-board NFs at scale | Implemented |
The All-in-One bash script bash script offers the minimal unattended instructions required for deploying a KRD in a node. It can be externally consumed with one single command.
curl -fsSL http://bit.ly/KRDaio | bash
The Environment Setup section provides the steps required to install KRD dependencies and define the Cluster setup before its provisioning.
Vagrant uses the default configuration to define the Virtual Machines used by the Kubernetes cluster, this configuration can be overwritten defining the config/pdf.yml file. The diagram tox environment generates the krd.png file to visualize the topology of the cluster.
tox -e diagram
The KRD command script provides an easy way to install complementary Kubernetes projects to an existing cluster. Those projects are grouped in KRD actions and it's possible to get the current list of current supported actions executing the following instruction:
./krd_command.sh -h
The actions which require the execution of a distributed commands were implemented through the install_k8s_addons KRD action. This action uses the KRD_ADDONS_LIST environment variable to specify the Ansible playbook to be executed.
KRD_ADDONS_LIST=virtlet ./krd_command.sh -a install_k8s_addons
Note: Some KRD AddOns have a corresponding validation script in the tests folder.
The functions defined in this project covers the lifecycle of a Kubernetes cluster. In other words, its possible to add more nodes, upgrade the existing deployment or remove the services. The following instruction shows how to upgrade the existing Kubernetes cluster to v1.18.10 using the Kubespray version v2.14.2:
KRD_KUBE_VERSION=v1.18.10 KRD_KUBESPRAY_VERSION=v2.14.2 ./krd_command.sh -a upgrade_k8s
| Name | Default | Description |
|---|---|---|
| KRD_DEBUG | false | Enables verbose execution |
| KRD_ANSIBLE_DEBUG | false | Enables ansible verbose execution |
| PKG_DEBUG | false | Enables cURL package installer verbose execution |
| KRD_KUBE_VERSION | v1.28.6 | Specifies the Kubernetes version to be upgraded |
| KRD_KUBESPRAY_VERSION | v2.24.1 | Specifies the Kubespray version to be used during the upgrade process |
| KRD_KUBESPRAY_REPO | https://github.com/kubernetes-sigs/kubespray |
Specifies the Git repository to fetch the Kubespray's source code |
| KRD_ENABLE_TESTS | false | Enables the functional tests during the deployment process |
| KRD_HELM_CHART | Specifies the Helm chart to be installed | |
| KRD_FOLDER | /opt/krd | KRD source code destination folder |
| KRD_ACTIONS_LIST | install_k8s | Specifies a list of KRD actions to be installed during the All-in-One execution |
| KRD_ADDONS_LIST | addons | Specifies the Kubernetes AddOns to be installed by install_k8s_addons |
| KRD_HUGEPAGES_ENABLED | true | Enable/Disable HugeTLB Pages on cluster nodes |
| KRD_QAT_PLUGIN_MODE | dpdk | Specifies the deployment mode for Intel QuickAssist Plugin |
| KRD_KUBESPHERE_DEVOPS_ENABLED | true | Enable/Disable KubeSphere DevOps System |
| KRD_KUBESPHERE_METRICS_SERVER_ENABLED | false | Enable/Disable Horizontal Pod Autoscaler on KubeSphere |
| KRD_KUBESPHERE_SERVICEMESH_ENABLED | false | Enable/Disable KubeSphere Service Mesh |
| KRD_ENABLE_ISTIO_ADDONS | false | Enable/Disable Istio AddOns(Grafana, Kiali and Prometheus) |
| KRD_METALLB_ADDRESS_POOLS_LIST | Specifies a list of L2 address pools for MetalLB configuration |
| Name | Default | Description |
|---|---|---|
| KRD_NETWORK_PLUGIN | flannel | Choose network plugin (calico, canal, cilium, contiv, flannel, weave) |
| KRD_FLANNEL_BACKEND_TYPE | host-gw | Type of flannel backend to use (vxlan, host-gw, udp) |
| KRD_CILIUM_TUNNEL_MODE | disabled | Encapsulation mode for communication between nodes (disabled, vxlan, geneve) |
| KRD_CILIUM_KUBE_PROXY_REPLACEMENT | probe | Specifies the type of kube-proxy replacement. |
| KRD_CILIUM_ENABLE_BPF_MASQUERADE | true | Enable/Disable native IP masquerade support in eBPF |
| KRD_CALICO_IPIP_MODE | Never | Configures Calico IP in IP encapsulation (Always, Never, CrossSubnet) |
| KRD_CALICO_VXLAN_MODE | Never | Configures Calico VXLAN encapsulation (Always, Never, CrossSubnet) |
| KRD_CALICO_NETWORK_BACKEND | bird | Configures Calico Network backend (bird, vxlan, none) |
| KRD_KUBE_PROXY_MODE | ipvs | Choose kube-proxy mode (iptables, ipvs) |
| KRD_DOWNLOAD_RUN_ONCE | true | Download images and binaries only once and then push them to the cluster nodes |
| KRD_DOWNLOAD_LOCALHOST | true | Make localhost the download delegate |
| KRD_MULTUS_ENABLED | false | Enable/Disable Kubernetes Multus CNI |
| KRD_CONTAINER_RUNTIME | containerd | Specifies the Container Runtime to be used for deploying kubernetes |
| KRD_DASHBOARD_ENABLED | false | Enable/Disable Kubernetes Dashboard |
| KRD_CERT_MANAGER_ENABLED | true | Enable/Disable Kubernetes Cert-Manager |
| KRD_INGRESS_NGINX_ENABLED | true | Enable/Disable NGINX Ingress Controller |
| KRD_KATA_CONTAINERS_ENABLED | false | Enable/Disable Kata Containers Runtime |
| KRD_CRUN_ENABLED | false | Enable/Disable crun |
| KRD_YOUKI_ENABLED | false | Enable/Disable youki |
| KRD_GVISOR_ENABLED | false | Enable/Disable gVisor |
| KRD_MANUAL_DNS_SERVER | Set to use a custom cluster DNS | |
| KRD_REGISTRY_MIRRORS_LIST | Specifies a list of additional registry mirrors | |
| KRD_INSECURE_REGISTRIES_LIST | Specifies a list of insecure-registries (IP address or domain name) | |
| KRD_LOCAL_VOLUME_PROVISIONER_ENABLED | true | Enable/Disable Local volume provisioner |
| KRD_DOCKER_VERSION | latest | Specifies the Docker version to be used for deploying Kubernetes |
| KRD_CONTAINERD_VERSION | 1.7.11 | Specifies the ContainerD version to be used for deploying Kubernetes |
| KRD_ENABLE_NODELOCALDNS | true | Enable/Disable NodeLocal DNSCache |
| KRD_NDOTS | 1 | Threshold for the number of dots which must appear in name resolution |
| KRD_RESOLVCONF_MODE | none | Specifies the DNS setup for non-k8s containers. |
| KRD_KUBE_PROXY_SCHEDULER | sh | Specifies the IPVS scheduling algorithm for allocating connections |
| KRD_METALLB_ENABLED | false | Enable/Disable MetalLB load-balancer |
| KRD_METALLB_ADDRESS_POOLS_LIST | Specifies a lists of Layer 2 Address pools for MetalLB | |
| KRD_KUBERNETES_AUDIT | false | Enable/Disable Auditing |
| KRD_KUBERNETES_AUDIT_WEBHOOK | false | Enable/Disable Audit Webhook |
| KRD_AUDIT_WEBHOOK_SERVER_URL | Audit Webhook server URL | |
| KRD_KUBELET_LOGFILES_MAX_NR | 5 | Maximum number of container log files that can be present for a container |
| KRD_KUBELET_LOGFILES_MAX_SIZE | 10Mi | Maximum size of the container log file before it is rotated |
This is an open project, several individuals contribute in different forms like coding, documenting, testing, spreading the word at events within others.
If you are interested in contributing, please refer to the code CONTRIBUTING guidelines.
Thanks to all the people who already contributed!
