Multus is no longer starting (infinite loop)

[ohault]

Summary

Using stable channel 1.28/stable, multus daemonset cannot be started meanwhile it can be started using 1.24/stable

Reproduction Steps

sudo snap install microk8s --classic
microk8s enable multus
=>
Waiting for multus daemonset to start................................

sudo snap install microk8s --classic --channel=1.24/stable
microk8s enable multus
=>
Waiting for multus daemonset to start................................
Multus is enabled
Multus is enabled with version:
multus-cni version:v3.4.2, commit:4eac660359f223d34bcaf0fddbc42fd542f02ba1, date:2020-05-15T12:43:46+0000

Introspection Report

microk8s kubectl describe pod -n kube-system

=>

Name: kube-multus-ds-9mnfc
Namespace: kube-system
Priority: 0
Service Account: multus
Node: pcwin11oha/172.27.218.10
Start Time: Sun, 24 Mar 2024 12:05:10 +0100
Labels: app=multus
controller-revision-hash=69c976674c
name=multus
pod-template-generation=1
tier=node
Annotations:
Status: Pending
IP: 172.27.218.10
IPs:
IP: 172.27.218.10
Controlled By: DaemonSet/kube-multus-ds
Init Containers:
install-multus-binary:
Container ID:
Image: ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9
Image ID:
Port:
Host Port:
Command:
cp
/usr/src/multus-cni/bin/multus
/host/opt/cni/bin/multus
State: Waiting
Reason: CreateContainerError
Ready: False
Restart Count: 0
Requests:
cpu: 10m
memory: 15Mi
Environment:
Mounts:
/host/opt/cni/bin from cnibin (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h56gq (ro)
Containers:
kube-multus:
Container ID:
Image: ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9
Image ID:
Port:
Host Port:
Command:
/entrypoint.sh
Args:
--multus-conf-file=auto
--multus-kubeconfig-file-host=/var/snap/microk8s/current/args/cni-network/multus.d/multus.kubeconfig
--cni-version=0.3.1
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 100m
memory: 50Mi
Requests:
cpu: 100m
memory: 50Mi
Environment:
Mounts:
/host/etc/cni/net.d from cni (rw)
/host/opt/cni/bin from cnibin (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h56gq (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
cni:
Type: HostPath (bare host directory volume)
Path: /var/snap/microk8s/current/args/cni-network/
HostPathType:
cnibin:
Type: HostPath (bare host directory volume)
Path: /var/snap/microk8s/current/opt/cni/bin/
HostPathType:
kube-api-access-h56gq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: :NoSchedule op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message

---

Normal Scheduled 5m7s default-scheduler Successfully assigned kube-system/kube-multus-ds-9mnfc to pcwin11oha
Normal Pulling 5m7s kubelet Pulling image "ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9"
Normal Pulled 4m34s kubelet Successfully pulled image "ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9" in 32.058s (32.058s including waiting)
Warning Failed 4m34s kubelet Error: failed to generate container "50566bcd392f86d00a35d4aded6520f1e32dc0f373ed68ec90aed66cf175a8e8" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount
Warning Failed 4m34s kubelet Error: failed to generate container "93dfaf569e709eb49cdb7219cbab9fa21043e8022aeb2648f738939c0220247a" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount
Warning Failed 4m21s kubelet Error: failed to generate container "c4c5f7b86be8da6eaf7f881a3de956b5f03c61984d28ae99feb4d09b8be73dab" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount
Warning Failed 4m8s kubelet Error: failed to generate container "0b1ecb344511f725ce7a0ad7fc8dad952d13aa63da1c852dcd96e2f1febc9984" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount
Warning Failed 3m53s kubelet Error: failed to generate container "717c88b449c5ff933b7a53f5f460339ddb382abf7d6170cdcd2d53a73ba927f1" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount
Warning Failed 3m41s kubelet Error: failed to generate container "8fc8635ed7ea0967fdab20745e0811ee8125a7b082037188cf4af56912dc6683" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount
Warning Failed 3m29s kubelet Error: failed to generate container "6541ec4495b46a00116b3a7af15fb554a2e0bb1dd478d64b10c87daf041c1418" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount
Warning Failed 3m14s kubelet Error: failed to generate container "0252bd0225b048d393b671fd46439c28a3b315ab5c8dc71702983f6944036b75" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount
Warning Failed 3m3s kubelet Error: failed to generate container "599cb58f63f08c8065aca17f902d908b20c1a846951b91177e95429894062b6b" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount
Warning Failed 2m23s (x3 over 2m49s) kubelet (combined from similar events): Error: failed to generate container "5cc95b135d7b09cbf3cedd6eb9f061a1aeb827cadc4706914dc65301f048b587" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount
Normal Pulled 6s (x22 over 4m34s) kubelet Container image "ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9" already present on machine

[neoaggelos]

Hi @ohault

What environment are you using? Is this perhaps on LXD? One thing you could try as a work-around is to edit the multus daemonset and remove the MountPropagation: Bidirectional line from https://github.com/canonical/microk8s-community-addons/blob/ec3dddc5a3b7efac45fe81daca35eba6b3b81ec0/addons/multus/multus.yaml#L160

[ohault]

I'm using WSL2.

Hi @ohault

What environment are you using? Is this perhaps on LXD? One thing you could try as a work-around is to edit the multus daemonset and remove the MountPropagation: Bidirectional line from https://github.com/canonical/microk8s-community-addons/blob/ec3dddc5a3b7efac45fe81daca35eba6b3b81ec0/addons/multus/multus.yaml#L160

I'm using WSL2.

[neoaggelos]

OK, thank you. Could you check whether the proposed workaround above helps?

[ohault]

OK, thank you. Could you check whether the proposed workaround above helps?

No so easy to test as multus.yaml is in ro in the /snap tree because I used snap to install microk8s.
Feel free to you tell me how I can test it ?
I can do it under WSL2/Ubuntu 20.4.6.

[neoaggelos]

OK, the addons themselves are read-write, and can be found under /var/snap/microk8s/common/addons. You could edit the manifest at /var/snap/microk8s/common/addons/community/addons/multus then enable the addon

[ohault]

OK, the addons themselves are read-write, and can be found under /var/snap/microk8s/common/addons. You could edit the manifest at /var/snap/microk8s/common/addons/community/addons/multus then enable the addon

It's exactly what I tried, but I got an error message like "The filesystem is read only".

[neoaggelos]

I think there must be a mixup here. The read-only parts of microk8s will be in /snap/microk8s/current/.... /var/snap/microk8s/common/... will not be readonly (it is actively used for microk8s to store and maintain the current state).

Can you try editing the file again? The command should be

sudo vim /var/snap/microk8s/common/addons/community/addons/multus/multus.yaml

Or can you share the exact error message that you are getting?

[ohault]

I have just tried from a fresh new system.
I confirm that on the new system, I have been able to modify the file /var/snap/microk8s/common/addons/community/addons/multus/multus.yaml
I don't understand why this was not the case, but it's a detail.
I confirm also that by removing the line "MountPropagation: Bidirectional" fix the multus starting issue.