Tainting Nodes to Fix Infrastructure Issues

Tainting Nodes on Kubernetes and OpenShift

Kubernetes allows you to taint nodes so that only certain pods can run on them.

This can allow you to do things such as set custom kubelet arguments, make sure all your pods land on a node with certain network/performance guarantees, and if nothing else, fence out other containers from competing with network and local disk bandwidth, which can be key performance bottlenecks.

Alpine DNS: Parallel resolution

Alpine DNS is parallelized, and thus if you have a wildcard resolver in your data center, the internal DNS names used by Black Duck's containers can get into a split-brain state, which will ultimately make your deployment non-viable. In Kubernetes versions after 1.7, hostAliases can be used to hardcode IP endpoints for well known hosts, such as kb.blackducksoftware.com, but it is difficult to use, because your service IP addresses aren't made until after you deploy Black Duck, and may change if you delete a service.

Alpine DNS Workaround : A use case for node tainting in Kubernetes

One way to prevent this is to inject your own resolv.conf file into each pod. (Note: There is pending upstream work in Kubernetes to enable this, although it is not done yet.) In the meantime, you could follow these steps:

• Taint a node to only run Black Duck containers: kubectl taint nodes mynode.mycompany.com blackduck=true:NoSchedule This will make sure that only Black Duck containers, and no other containers, run on this node. That way, any modifications you make to the node won't effect the behavior of other pods on the cluster.
• Add tolerations to Black Duck deployment pods for blackduck=true.

How do I make sure that pods schedule to my tainted node?

You'll need to add the following to your pod definitions. Since every container needs to talk to something else on the internal Kubernetes network, you'll need to do this for every pod.

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  ...
spec:
  replicas: 
      ...
  template:
    metadata:
      ...
    spec:
      nodeSelector:
         ...  
      tolerations:
        key: blackduck
        operator: Equal
        value: true

• Modify your kubelet startup flags to inject a custom resolv.conf, for example:

KUBELET_ARGS="--pod-manifest-path=/etc/kubernetes/manifests 
--cadvisor-port=0 
--pod-infra-container-image=gcr.io/google_containers/pause-amd64:3.0 
--node-status-update-frequency=10s 
--docker-disable-shared-pid=True 
--client-ca-file=/etc/kubernetes/ssl/ca.pem 
--tls-cert-file=/etc/kubernetes/ssl/node-k8s-master-1.pem 
--tls-private-key-file=/etc/kubernetes/ssl/node-k8s-master-1-key.pem 
--anonymous-auth=false 
--read-only-port=10255 
--cgroup-driver=cgroupfs 
--cgroups-per-qos=True 
--fail-swap-on=True 
--enforce-node-allocatable="" --cluster-dns=10.233.0.3 --cluster-domain=kubernetes.tdlab --resolv-conf=/etc/resolv.conf

In your resolv.conf, you can entirely remove your wildcard DNS settings. To see what a custom resolv.conf should look like, copy it out of a running pod, and simply delete the lines referencing your data center's wildcard DNS.

This will ensure that all your Black Duck containers land only on the node which you have tainted.

Don't I still need to resolve outside IP addresses to talk to Black Duck's Knowledge Base (KB)?

After this, if some external hosts like kb.blackducksoftware.com aren't resolved in containers, you can modify hostAliases to manually inject them into etc/hosts of your containers.