Bare Metal Kubernetes

We see this project both as a proof of concept, but also as a lab and a great way to learn the basics of operating a Kubernetes (k8s) cluster.

TL;DR; - getting things running

• - Set up MaaS and hardware
• - Install python3, virtualenv and virtualenvwrapper
• - Install requirements for this project in a virtualenv
• - Set environment variables for MAAS:
  • MAAS_API_URL
  • MAAS_API_KEY
• To run iteratively and measure results:
  • use the ./measure.py python script
• To run process from end to end without timing/etc:
  • use the ./_run.sh bash script

Why bother?

It's important to start from the position of understanding 'why'. In our case, we had a bunch of hardware sitting around and it felt like a waste to let it just sit there, but additionally, a number of our clients have pretty much the same situation - hardware they already own (or lease, or otherwise are already paying for) that may not make for the best production K8s system in the world, but it's a great way to learn and even run development environment(s) from.

Resources

There are a couple of great walkthroughs focused on bare metal K8s that use a myriad of techniques:

• Josh Rendek's awesome walkthrough
• Alex Ellis' (of OpenFaaS fame) walkthrough
• Kubernetes' walkthroughs:
  • Fedora - Single Node and Multi Node
  • Ubuntu - LXD, Juju, Rancher

These are ALL great resources. We have a few specific capabilities we're shooting for that we outline on a per-section basis.

We broke the process into a few main chunks:

• 'Lights Out' to Operating System 'up and running'
• Base OS to 'ready for K8s'
• K8s installation
• K8s Basic Configuration

Before we start...

There were some core technology choices we made in getting this 'thing' put together.

• Cannonical's Metal as a Service for hardware provisioning/management
  • We'll share more on this in the hardware-level management layer section below
• Ansible for configuration management
  • Running remote commands with a way to share data centrally was critical to coordinating cluster deployment(s)
  • We also ended up leveraging a custom dynamic inventory script so we could use tags within MaaS to identify cluster roles.
• Python to glue it together...
  • Ansible is based on Python... and there are a few API clients pre-baked for Python MaaS as well..
  • It also offers some cool opportunities to generate configurations for our networking gear, etc.
  • We use virtualenv and virtualenvwrapper to manage our python environment(s) locally:
    • mkvirtualenv -p python3 maas

Hardware-level management layer - Ubuntu Metal as a Service

Our first set of requirements looks a lot like that of what public cloud Infrastructure as a Service (IaaS) providers offer:

• API driven deployment and re-deployment of 'instances'
• API or at least 'HTTP accessible' power management
• Ability to handle entire 'cold start' hardware deployment
  • 'Completely off' (no power) to up and running

We looked at a few options and ended up with using Cannonical's Metal as a Service (MaaS) to deploy Ubuntu on hardware via PXE Boot.

• CoreOS' Matchbox solution is pretty cool too...
  • We were looking for as few 'moving parts' as possible in the early stage and Matchbox doesn't quite hit all the 'cold boot' checkboxes we'd have liked.
  • This might work really well in a VMWare-based environment where you put the instance provisioning process in front of Matchbox in the workflow

MaaS also offers a more 'elastic' setup where it can manage KVM instances in 'Pods'. Want to really move fast? Check out how to set up pods in their tutorial. We were even able to this out on machines we provisioned via MaaS!

Prerequisites/Setting up MaaS

We're assuming that the MaaS basic setup documentation is 'enough' to get you moving. That being said, the minimum viable setup to have 3 k8s nodes (1 master, 2 nodes) is something along the lines of:

• 3 machines that are configured for network boot via PxE (Bios/Boot setting)
  • No joke, our first setup used 3 'old' Dell OptiPlex 7010 desktops
• 1 machine to run MaaS
  • Again, no joke, we used a Zotac ZBOX CI323... we've since set up MaaS properly :)
  • hey.. it has 2 nics (which we bonded) and we dumped an SSD in it... it works.
• A supported power management device (or IPMI-enabled server)
  • Since we didn't use servers, we picked up some Digital Loggers PDUs... they work really well for us!
    • We've tried both their Web Power Switch and Universal Input PDU
• Since MaaS acts as the DHCP server, you'll want to configure your network so that this can happen.
  • We just set up a separate VLAN and set it up without a DHCP server ...
• If you can 'deploy' a machine and you get a functioning Ubuntu server, you're ready to actually get started.

Here's what our basic setup looks like:

Hardware Provisioning

Assuming that you've got MaaS up and running, we'll need to get a few things configured:

• Set up your python environment (virtualenv and virtualenvwrapper are assumed):
  • pip install -r requirements.txt
• Go grab your MaaS API key and host/url details - it's used by the Ansible dynamic inventory script to hit the MaaS API.
  • Our maas file pulls values from the environment for MAAS_API_KEY and MAAS_API_URL
• In MaaS, tag your machines with our 'arbitrary' (semi-configurable in the maas file) tags:
  • k8s-master for your (minimally) single master
  • k8s-node for the rest of the machines you're setting up.

Cycling/testing the hardware-level provisioning looks a little something like this:

export MAAS_API_URL=http://172.16.16.2:5240/MAAS
export MAAS_API_KEY=apikeygoeshere
./maas --reset

Output from the --reset command will be released and then deployed.

5-6 (ish) minutes later, you should see something like this:

(maas) pmdev13:k8s-baremetal pmdev$ ./maas --reset
Starting reset process at 2018-10-09 20:48:44.443620
Releasing host - dev-09
Releasing host - dev-10
Releasing host - dev-11
Hosts have reached Ready state: ['dev-09', 'dev-10', 'dev-11']
Deploying Host - dev-09
Deploying Host - dev-10
Deploying Host - dev-11
0 of 3 hosts ready to deploy
0 of 3 hosts ready to deploy
0 of 3 hosts ready to deploy
1 of 3 hosts ready to deploy
Hosts have reached Deployed state: ['dev-09', 'dev-10', 'dev-11']
Process complete at 2018-10-09 20:55:04.924346
MaaS cycle time - 380.480726 seconds

Some cycle times from testing while writing this document:

Ansible Playbook Sequencing

Base OS to 'Ready for k8s'

Everything to prep the machines to get them ready for Kubernetes is contained within the k8s-01-base.yaml playbook.

Once we have an OS, setting things up to run Kubernetes is ... pretty straightforward. Since we're using Ansible as our configuration management tool, the process of getting things moving are easy enough to follow:

• Add a few specific repositories for packages and tools we'll need to install
  • Docker
  • Kubernetes
• Actually install programs and tools including:
  • kubeadm
  • kubelet
  • docker-ce
• Remove and disable swap per kubelet requirements
• install helm binaries on master

K8s Installation

The k8s-02-install.yaml playbook handles actually initializing the kubernetes cluster master and nodes. We use kubeadm to get that done. Of specific import, our configuration is a bit opinionated:

• Single master cluster
• The first IP assigned to the master (in MaaS via the MaaS API in our dynamic inventory) is used to configure the master API address
• We don't add a ton of special configs... but all configs are in the vars/k8s_config.yaml
• We do apply a few very basic manifests:
  • Weave network overlay
  • Service account for Helm Tiller to use
  • Then we run helm init to make sure we're ready to deploy more stuff!
• While it's a little 'hack-ey' we also write out a variable file at vars/k8s_master.yaml that contains the master IP so we can use it in downstream playbooks

K8s Basic Configuration

The k8s-03-install.yaml playbook installs a bunch of stuff to the kube-system namespace:

• Storage Class setup for NFS servers (we have 2 for dev purposes...)
  • We use the stable/nfs-client-provisioner helm chart
• nginx-ingress to handle inbound connections bound to the master node's IP address
• Monitoring charts
  • Prometheus
  • Grafana

Local Access Configuration

The k8s-04-localhost.yaml playbook grabs a kubeconfig file from the master, copies it locally and then merges any existing config file before placing it in ~/.kube/config.