Terraform Provisioner for a 3 node Consul Service Discovery cluster on top of Kubernetes running on ARM architecture / Raspberry Pi. Makes use of custom Kubernetes Terraform Provider from https://github.com/sl1pm4t/terraform-provider-kubernetes. Will ultimately change this to use the standard Terraform Kubernetes provider when StatefulSets are supported. See related GitHub Issue.
Includes Terraform resource definitions for Kubernetes Service, Persistent Volume (for Consul data persistent storage), and Stateful Set. Functionality mostly similar to deploying using a Helm chart, but currently no version of Tiller available to run on ARM architecture. This method allows for single-command Terraform deployment, with a single variables config file.
Container based off of Docker image located at https://hub.docker.com/r/clayshek/consul-arm/. Runs on top of ARM32v7/Debian:stretch-slim, approx size 96 MB. Containers based off this image configurable via terraform.tfvars and should work for most use-cases, but for deeper configuration needs, custom Docker images can be built and used by modifying image_name parameter in terraform.tfvars.
Makes use of NFS shares as Kubernetes Persistent Volumes for Consul data storage.
Note that this deployment is not to be considered "production ready" as there are some small issues and several items that would not be considered best-practice, including in terms of security. However, this is a good starting point for a lab-based Consul setup.
Some code based off of: https://github.com/hashicorp/docker-consul/ & https://github.com/kelseyhightower/consul-on-kubernetes
Native Kubernetes manifest files for creating Service, PersistentVolumes, and StatefulSet with volumeClaimTemplates are included in the _non-tf-k8s-manifests folder as an alternative to Terraform deployment. This is less flexible than Terraform deployment, but an alternative regardless.
- Working Kubernetes cluster (developed on v1.9) on Raspberry Pi. Info
- Terraform (tested with v0.11.7)
- Terraform must have connectivity to Kubernetes. Either run on K8s master (easiest), or other workstation with properly configured Kube config file and Terraform provider setup.
- Pre-configured NFS shares (default) or other Kubernetes supported Persistent Volume for Consul data storage. If using other than NFS, customization of main.tf likely required in addition to tfvars file. See Terraform kubernetes_persistent_volume resource documentation for further details. An alternative, if not using persistent storage, would be to comment out all Persistent Volume related code, but risk Consul data loss. See NFS Setup section below for more details.
Will not go in depth on NFS setup here, but as this repo is coded, it requires only two variables in terraform.tfvars:
- nfs_server - IP address or resolvable hostname
- vol_path - The path of the shares, as should be visible using
showmount -e IP_ADDRESS
The single vol_path variable will support any number of Consul servers (consul_count variable), so long as the NFS shares for each exist, and so long as each share ends in a number beginning with zero and incrementing by one. Example /etc/exports file for my deployment of 3 Consul servers with associated NFS shares on an attached USB drive:
/mnt/usbdrive1/consul-data0 192.168.1.0/24(rw,sync)
/mnt/usbdrive1/consul-data1 192.168.1.0/24(rw,sync)
/mnt/usbdrive1/consul-data2 192.168.1.0/24(rw,sync)
NFS References:
- Clone the repository
- Customize the parameters in the terraform.tfvars file as applicable for provisioning.
- Build the custom Terraform Kubernetes provider (see steps below), or alternatively use the pre-built binary (terraform-provider-kubernetes.exe for Windows or terraform-provider-kubernetes for ARM) included in this repo.
- Run
terraform init(required for first run). - Apply the configuration:
terraform apply
- Remove the configuration from Kubernetes:
terraform destroy
As described in the Summary section above, the Terraform Kubernetes provider does not support StatefulSet deployments. Until that feature is added (GitHub Issue, another one), a custom TF K8s Provider with that capability is used: https://github.com/sl1pm4t/terraform-provider-kubernetes/tree/custom
In order to use that provider with the code in this repo, follow the steps below to clone & build:
- Install Go: https://golang.org/doc/install
- Clone Repo & Build Provider
$ mkdir -p $GOPATH/src/github.com/sl1pm4t
$ cd $GOPATH/src/github.com/sl1pm4t
$ git clone -b custom https://github.com/sl1pm4t/terraform-provider-kubernetes.git
$ cd terraform-provider-kubernetes
$ make build (or go build on windows)
- Copy the compiled provider (either terraform-provider-kubernetes.exe on Windows, or terraform-provider-kubernetes on Linux) to same directory as the terraform code.
Reference: Writing Custom Terraform Providers
- Metrics & Monitoring.
- Enable Encryption: https://www.consul.io/docs/agent/encryption.html
- Change to use Terraform standard provider when StatefulSet is supported. See Git Issue
- Evaluate other Storage options. Possibly Rook? Ceph?
- Terraform destroy / Kubectl delete operation does not reliably remove Kubernetes Persistent Volume Claims, requires removal with
kubectl delete pvc PVC-NAME. Likely something to do with the way volumeClaimTemplate provisions these. Occurs with Terraform or with native K8s manifests. Dig into this and address.
This is open-sourced software licensed under the MIT license.