Krustlet on Kind shows some weird behavior

[deepu105]

I followed the quick start for kinD and installed krustlet. During the install step I got some errors on the console and when trying to deploy the hello-wasm sample I get error

Running the below command gives some error on the console but I could see Krustlet as a node but the Roles field was empty

KUBECONFIG=~/.krustlet/config/kubeconfig \
  ./krustlet-wasi \
  --node-ip=172.17.0.1 \
  --node-name=krustlet \
  --bootstrap-file=${HOME}/.krustlet/config/bootstrap.conf

Console out, I believe this comes from kindnet pod being scheduled on krustlet and failing since its not a wasm workload

BOOTSTRAP: TLS certificate requires manual approval. Run kubectl certificate approve fedora-tls
BOOTSTRAP: received TLS certificate approval: continuing
Jan 12 13:07:15.313 ERROR kubelet::state::common::image_pull: error=unsupported media type: application/vnd.docker.distribution.manifest.list.v2+json
Jan 12 13:07:26.374 ERROR kubelet::state::common::image_pull: error=unsupported media type: application/vnd.docker.distribution.manifest.list.v2+json
Jan 12 13:07:47.436 ERROR kubelet::state::common::image_pull: error=unsupported media type: application/vnd.docker.distribution.manifest.list.v2+json
Jan 12 13:08:28.563 ERROR kubelet::state::common::image_pull: error=unsupported media type: application/vnd.docker.distribution.manifest.list.v2+json
Jan 12 13:09:49.656 ERROR kubelet::state::common::image_pull: error=unsupported media type: application/vnd.docker.distribution.manifest.list.v2+json
Jan 12 13:12:30.617 ERROR kubelet::state::common::image_pull: error=unsupported media type: application/vnd.docker.distribution.manifest.list.v2+json
Jan 12 13:17:31.550 ERROR kubelet::state::common::image_pull: error=OCI API error: authentication required on https://registry-1.docker.io/v2/kindest/kindnetd/manifests/v20210326-1e038dc5
Jan 12 13:22:32.503 ERROR kubelet::state::common::image_pull: error=OCI API error: authentication required on https://registry-1.docker.io/v2/kindest/kindnetd/manifests/v20210326-1e038dc5

Nodes

❯ kubectl get nodes -o wide
NAME                 STATUS   ROLES                  AGE   VERSION         INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION            CONTAINER-RUNTIME
kind-control-plane   Ready    control-plane,master   27m   v1.21.1         172.21.0.2    <none>        Ubuntu 21.04   5.15.12-200.fc35.x86_64   containerd://1.5.2
krustlet             Ready    <none>                 15m   1.0.0-alpha.1   172.17.0.1    <none>        <unknown>      <unknown>                 mvp

Then I tried deploying a pod

kubectl apply -f - <<EOF                                           
apiVersion: v1
kind: Pod
metadata:         
  name: hello-wasm
spec:        
  containers:       
  - name: hello-wasm                           
    image: webassembly.azurecr.io/hello-wasm:v1
  tolerations:       
  - effect: NoExecute      
    key: kubernetes.io/arch
    operator: Equal                                                           
    value: wasm32-wasi   # or wasm32-wasmcloud according to module target arch
  - effect: NoSchedule     
    key: kubernetes.io/arch
    operator: Equal   
    value: wasm32-wasi
EOF

The pod status is ExistCode:0 which I assume is expected for this sample. If I look at the logs I can see the output is printed. And when using a sample that runs something in loop I can see running status. SO I guess that's all normal.

But on kube-system namespace The kindnet-* pod gets scheduled on the krustlet node and the already existing pod on the default node goes into a CrashLoopBackOff. Not sure if this is an issue or something expected

Also on the krustlet-wasi logs I see this repeatedly, I believe from trying to run the kindnet docker image on krustlet

Jan 13 06:01:07.268 ERROR kubelet::state::common::image_pull: error=OCI API error: authentication required on https://registry-1.docker.io/v2/kindest/kindnetd/manifests/v20210326-1e038dc5

kind v0.11.1 go1.16.5 linux/amd64
krustlet 1.0.0-alpha.1
OS : Fedora 35

Will be happy to provide more info if needed

[deepu105]

The original issues I face seems to be from some minor inconsistencies in docs that I probably misunderstood. I created a PR to the docs projects to propose some updates krustlet/krustlet-docs#20