Skip to content

Kubernetes controller for labelling a node with devicetree properties

License

Notifications You must be signed in to change notification settings

adaptant-labs/k8s-dt-node-labeller

Repository files navigation

Device Tree Kubernetes Node Labeller

This tool provides a custom Kubernetes controller for automatically labelling nodes with devicetree properties.

It is inspired by and re-uses much of the Kubernetes controller plumbing from the amdgpu-node-labeller.

Use Cases

k8s-dt-node-labeller was developed in order to facilitate targeted deployments into hybrid (e.g. armhf, arm64) Kubernetes clusters, as well as for targeting heterogeneous accelerators in Edge deployments - specifically those that show up as platform devices (FPGAs, embedded GPUs, etc.) as opposed to those that are dynamically discoverable at run-time via bus enumeration (e.g. USB, PCI). The latter cases are handled by the official node-feature-discovery (NFD) labeller, which can be used in conjunction with k8s-dt-node-labeller for more comprehensive node labelling.

Overview

k8s-dt-node-labeller is primarily concerned with top-level system characteristics, and therefore limits itself to walking through the top-level devicetree node looking for compatible properties to expose as Kubernetes node labels. As k8s-dt-node-labeller requires access to the target node's devicetree instance, the labeller must be run directly on the target node that is to be labelled. An overview of the general labelling flow (in the context of a Kubernetes DaemonSet) is provided below:

Overview Diagram

It is expected that the node labeller is run when the node is registered with the cluster, such that pods relying on specific labels are able to be scheduled and placed appropriately.

Labelling is possible in a number of different ways:

  • Running Node-Local via the CLI App
  • Running Node-Local via Docker
  • Deploying as a Kubernetes DaemonSet directly
  • Deploying as part of a combined Kubernetes NFD DaemonSet

Regardless of the method of deployment, the labels will be generated and applied to the local node.

Limitations

k8s-dt-node-labeller only generates labels from properties at the top-level of the devicetree and those specifically defined by the user by design. It does not recurse down the tree, nor does it make any attempt to enumerate busses that are better served by bus-specific device detection logic. While the labeller can attest to the existence of a node in the devicetree, it offers no device-specific information or control - all of which would need to be implemented through a device-specific device plugin.

Installation

If planning to use the CLI App directly, this can be compiled and installed as other Go projects:

$ go get github.com/adaptant-labs/k8s-dt-node-labeller

For Docker containers and Kubernetes deployment instructions, see below.

Usage

General usage is as follows:

$ k8s-dt-node-labeller --help
devicetree Node Labeller for Kubernetes
Usage: k8s-dt-node-labeller [flags] [-n devicetree nodes...]

  -d	Display detected devicetree nodes
  -f	Write detected features to NFD features file
  -kubeconfig string
    	Paths to a kubeconfig. Only required if out-of-cluster.
  -n string
    	Additional devicetree node names

A dry-run is possible by specifying the -d flag:

$ k8s-dt-node-labeller -d
Discovered the following devicetree properties:

beta.devicetree.org/nvidia-jetson-nano: 1
beta.devicetree.org/nvidia-tegra210: 1

By default, compatible strings from the top-level / node are discovered and converted to node labels. Additional node specifications are possible via the -n flag, as below:

$ k8s-dt-node-labeller -d -n gpu pwm-fan
Discovered the following devicetree properties:

beta.devicetree.org/nvidia-jetson-nano: 1
beta.devicetree.org/nvidia-tegra210: 1
beta.devicetree.org/nvidia-tegra210-gm20b: 1
beta.devicetree.org/nvidia-gm20b: 1
beta.devicetree.org/pwm-fan: 1

In the case of Docker or Kubernetes Deployment, note that these arguments can also be passed on to the container directly.

Running Node-Local via the CLI App

In order to submit the labels to the cluster, ensure a valid kubeconfig can be found (e.g. by setting the KUBECONFIG environment variable, or through specifying the location with the -kubeconfig parameter) and execute the node labeller as-is:

$ k8s-dt-node-labeller 
{"level":"info","ts":1585701002.694782,"logger":"k8s-dt-node-labeller.entrypoint","msg":"setting up manager"}
{"level":"info","ts":1585701003.1179338,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1585701003.118381,"logger":"k8s-dt-node-labeller.entrypoint","msg":"Setting up controller"}
{"level":"info","ts":1585701003.1185818,"logger":"k8s-dt-node-labeller.entrypoint","msg":"starting manager"}
{"level":"info","ts":1585701003.1190712,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1585701003.1193638,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"k8s-dt-node-labeller","source":"kind source: /, Kind="}
{"level":"info","ts":1585701003.2218263,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"k8s-dt-node-labeller"}
{"level":"info","ts":1585701003.222448,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"k8s-dt-node-labeller","worker count":1}
...

Running Node-Local via Docker

Multi-arch Docker images are available on Docker Hub at adaptant/k8s-dt-node-labeller. These may be run as-is in-cluster, or out of cluster with an appropriate KUBECONFIG passed through.

Note that as the labeller requires access to the host's /sys/firmware directory, this must either be passed through explicitly, or the container run in privileged mode:

$ docker run --privileged adaptant/k8s-dt-node-labeller
...

Deploying as a Kubernetes DaemonSet

An example deployment configuration for a DaemonSet is provided in k8s-dt-labeller-ds.yaml, which can be directly applied to the running cluster:

$ kubectl apply -f https://raw.githubusercontent.com/adaptant-labs/k8s-dt-node-labeller/k8s-dt-labeller-ds.yaml

This will create a special dt-labeller service account, cluster role, and binding with the permission to list and reconcile nodes. Note that as the labeller requires access to an unmasked /sys/firmware, it must also be run in a privileged securityContext.

Deploying with NFD

When used together with NFD k8s-dt-node-labeller runs in one-shot mode as an init container, providing a static list of discovered features and labels to pass off to NFD, for use with its local feature discovery.

$ kubectl apply -f https://raw.githubusercontent.com/adaptant-labs/k8s-dt-node-labeller/k8s-dt-labeller-nfd.yaml

In this configuration, the nfd-master, nfd-worker, and k8s-dt-node-labeller are all deployed in the same Pod in order to facilitate node-local labelling. While the k8s-dt-node-labeller requires access to an unmasked /sys/firmware, and must, therefore, run in a privileged securityContext, the NFD application containers are able to run without elevated permissions.

Verifying Labelling

After the labeller has run, the node labels can be viewed from kubectl:

$ kubectl describe node jetson-nano
Name:               jetson-nano
Roles:              <none>
Labels:             beta.devicetree.org/nvidia-jetson-nano=1
                    beta.devicetree.org/nvidia-tegra210=1
                    beta.kubernetes.io/arch=arm64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    k3s.io/hostname=jetson-nano
                    k3s.io/internal-ip=192.168.xxx.xxx
                    kubernetes.io/arch=arm64
                    kubernetes.io/hostname=jetson-nano
                    kubernetes.io/os=linux
                    node.kubernetes.io/instance-type=k3s
                    ...

Features and bugs

Please file feature requests and bugs in the issue tracker.

Acknowledgements

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 825480 (SODALITE).

License

k8s-dt-node-labeller is licensed under the terms of the Apache 2.0 license, the full version of which can be found in the LICENSE file included in the distribution.