v0.21.0: kind container makes changes to binfmt configutation

[f84anton]

What happened:
I'm using orbstack on macbook m1, upgraded to kind 0.21.0 and created a new cluster with k8s 1.23, image kindest/node:v1.23.17@sha256:fbb92ac580fce498473762419df27fa8664dbaa1c5a361b5957e123b4035bdcf
After that I've got problem with running amd64 images in docker or inside kind:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
exec /service: exec format error

config file:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  TTLAfterFinished: true
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."localhost:5000"]
    endpoint = ["http://kind-registry:5000"]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."kind-registry:5000"]
    endpoint = ["http://kind-registry:5000"]
kubeadmConfigPatches:
- |
  apiVersion: kubelet.config.k8s.io/v1beta1
  kind: KubeletConfiguration
  evictionHard:
    nodefs.available: "1%"
    imagefs.available: "1%"
    nodefs.inodesFree: "1%"
    memory.available: "50Mi"
  imageMinimumGCAge: "1h"
  imageGCHighThresholdPercent: 90
  imageGCLowThresholdPercent: 85
  allowedUnsafeSysctls:
  - "net.*"
nodes:
- role: control-plane
  extraPortMappings:
  - containerPort: 30080
    hostPort: 80
  - containerPort: 30443
    hostPort: 443

Here is how I created a cluster:

kind create cluster \
 --name kind \
 --image  "kindest/node:v1.23.17@sha256:fbb92ac580fce498473762419df27fa8664dbaa1c5a361b5957e123b4035bdcf"\
 --config /path/to/config \
 --wait 5m`

Here is binfmt info after kind container was created:

%  docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64"
  ],
  "emulators": [
    "python3.11"
  ]
}

After kind cluster delete and orbstack restart I see that the list goes back to normal:

%  docker run --privileged --rm tonistiigi/binfmt
{
  "supported": [
    "linux/arm64",
    "linux/amd64",
    "linux/riscv64",
    "linux/ppc64le",
    "linux/s390x",
    "linux/386",
    "linux/mips64le",
    "linux/arm/v7",
    "linux/arm/v6"
  ],
  "emulators": [
    "mac-macho-arm64",
    "mac-macho-x86_64",
    "mac-universal-arm64",
    "mac-universal-x86_64",
    "qemu-arm",
    "qemu-i386",
    "qemu-mips64el",
    "qemu-ppc64le",
    "qemu-riscv64",
    "qemu-s390x",
    "qemu-x86_64",
    "rosetta",
    "rosetta",
    "rosetta"
  ]
}

I had to roll back to 0.20.0, there is no issue with this version and k8s image kindest/node:v1.23.17@sha256:59c989ff8a517a93127d4a536e7014d28e235fb3529d9fba91b3951d461edfdb

What you expected to happen:
I've expected that I will not loose ability to run amd64 images.

How to reproduce it (as minimally and precisely as possible):
Create kind cluster with v0.21.0 on macbook m1 with orbstack.

Anything else we need to know?:
there is a similar issue for minikube kubernetes/minikube#17700

Environment:

• kind version: (use kind version): kind v0.21.0 go1.20.13 darwin/arm64
• Runtime info: (use docker info or podman info):

% docker info
Client:
 Version:    25.0.2
 Context:    orbstack
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /Users/antonfaizrakhmanov/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.3
    Path:     /Users/antonfaizrakhmanov/.docker/cli-plugins/docker-compose
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.17.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-scan

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 13
 Server Version: 25.0.2
 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8c087663b0233f6e6e2f4515cee61d49f14746a8
 runc version: 82f18fe0e44a59034f3e1f45e475fa5636e539aa
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.13-orbstack-00122-g57b8027e2387
 Operating System: OrbStack
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 2.513GiB
 Name: orbstack
 ID: 70e419d6-93c9-4b9a-bb82-5507a43455ef
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
 Default Address Pools:
   Base: 192.168.215.0/24, Size: 24

• OS (e.g. from /etc/os-release): macos 14.2.1 (23C71) MacBook Air M1 2020
• Kubernetes version: (use kubectl version):

% kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:51:05Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.17", GitCommit:"953be8927218ec8067e1af2641e540238ffd7576", GitTreeState:"clean", BuildDate:"2023-06-15T02:24:51Z", GoVersion:"go1.19.6", Compiler:"gc", Platform:"linux/arm64"}

[afbjorklund]

Same ubuntu python package issue as in minikube

• Starting Minikube messes up binfmt config for Docker, making it unable to run cross-architecture images kubernetes/minikube#17700 (comment)

[BenTheElder]

Thanks for the pointer @afbjorklund, that's a pretty annoying change.

Might look at disabling systemd-binfmt entirely, but just removing the python file should be an OK stopgap.

[BenTheElder]

This test image has 1.23.17 with #3511 bentheelder/kind-node:zz-binfmt-test@sha256:a388d0f1ae8f894e77b6b80ad2bce7f3457f9acd29c90001dae7c9d9d6ffc564

[aojea]

:/

[embik]

This test image has 1.23.17 with #3511 bentheelder/kind-node:zz-binfmt-test@sha256:a388d0f1ae8f894e77b6b80ad2bce7f3457f9acd29c90001dae7c9d9d6ffc564

@xmudrii and I tested this image and it worked for us!

[BenTheElder]

we'll move forward with that proposed fix and then release it + new images soonish, just sent the PR to the merge queue

[potiuk]

Cool. We just experienced the same and tried to downgrade to kind 0.20 but it did not help. We have a workaround for now (k9s image does not work nicely but we can still use brew locally installed k9s) but it would be great to get it fixed .

[BenTheElder]

Cool. We just experienced the same and tried to downgrade to kind 0.20 but it did not help. We have a workaround for now (k9s image does not work nicely but we can still use brew locally installed k9s) but it would be great to get it fixed .

You'd have to use an image from 0.20.0 release (including @sha256 digest to be sure), the change will be purely in the node image. (also those images should work with 0.21.0, but are old and may have other since-fixed issues).

I don't think 0.20.0 images had systemd-binfmt but haven't confirmed yet.

[BenTheElder]

The fix is merged, but Kubernetes has monthly patch releases slated for tomorrow, so it probably makes more sense to cut the kind release afterwards.

In the meantime, these images have the fix and would've been in a release today:

• v1.29.1: kindest/node:v1.29.1@sha256:0c06baa545c3bb3fbd4828eb49b8b805f6788e18ce67bff34706ffa91866558b
• v1.28.6: kindest/node:v1.28.6@sha256:e9e59d321795595d0eed0de48ef9fbda50388dc8bd4a9b23fb9bd869f370ec7e
• v1.27.10: kindest/node:v1.27.10@sha256:e6b2f72f22a4de7b957cd5541e519a8bef3bae7261dd30c6df34cd9bdd3f8476
• v1.26.13: kindest/node:v1.26.13@sha256:8cb4239d64ff897e0c21ad19fe1d68c3422d4f3c1c1a734b7ab9ccc76c549605
• v1.25.16: kindest/node:v1.25.16@sha256:e8b50f8e06b44bb65a93678a65a26248fae585b3d3c2a669e5ca6c90c69dc519
• v1.24.17: kindest/node:v1.24.17@sha256:bad10f9b98d54586cba05a7eaa1b61c6b90bfc4ee174fdc43a7b75ca75c95e51
• v1.23.17: kindest/node:v1.23.17@sha256:14d0a9a892b943866d7e6be119a06871291c517d279aedb816a4b4bc0ec0a5b3

[potiuk]

No worries - we can wait :) . Thanks for the quick fix !

[BenTheElder]

https://github.com/kubernetes-sigs/kind/releases/tag/v0.22.0