K8s Cluster

Features

• Lots of self-hosted services
• Flux GitOps with this repository (kubernetes directory)
• Ansible node provisioning and K3s setup (Ansible roles and playbooks)
• SOPS secrets stored in Git
• Renovate bot dependency updates
• WireGuard VPN pod gateway via paid service
• WireGuard VPN proxy hosted on VPS
• Cloudflared HTTP tunnel
• K8s gateway for local DNS resolution to the cluster and NGINX ingress controller
• Both internal & external services with a service gateway
• OIDC authentication with LDAP
• Automatic Cloudflare DNS updates (ddns cronjob)
• Cilium container networking interface (CNI) and layer 4 loadbalancing
• ZFS
• JBOD mergerfs union NFS with SnapRAID backup for low-touch media files (snapraid-runner kubernetes cronjob)
• Restic backups to remote and local buckets (backup namespace)
• go-task shorthand for useful commands (Taskfile and taskfiles)

Usage

Setup and usage is inspired heavily by this homelab gitops template and the k8s-at-home community. You can find similar setups with the k8s at home search. Historical revisions of this repository had rootless Podman containers deployed with ansible as systemd units, and a single-node docker compose orchestration before that.

Looking for a simpler devops experience? Checkout my docker deployment at brettinternet/homelab.

Setup

Dependencies

Install go-task

Install dependencies and setup environment:

task init

Provision

Then, provision your infrastructure:

task ansible:{init,list,ping,setup,install,status}

DNS and Tunnel

Setup a Cloudflare Tunnel.

cloudflared tunnel login
cloudflared tunnel create cluster

Add the tunnel's credentials.json to the value in cloudflared-secret and tunnel ID to cluster-secrets.sops.yaml.

Add a Cloudflare API token with these permissions to the value in external-dns-secret.

• Zone - DNS - Edit
• Account - Cloudflare Tunnel - Read

Deploy

Kubernetes

Verify flux can be installed. Then, push changes to remote repo and install.

task flux:{verify,install}

Push latest to repo - you can use the wip.sh script for that with task wip.

task flux:reconcile

task kubernetes:resources

Most deployments in this repo use an app-template chart with these configuration options.