Starting with certain models introduced in late 2020, Apple began the transition from Intel processors to Apple silicon in Mac computers. These CPUs have ARM architecture and are incompatible with Intel processors.
Developing transforms for either the python or Ray runtimes, without KubeFlow pipelines (KFP), should have no issues on Apple silicon Macs, or other platforms for that matter. Therefore, to the extent the supported versions of python are used, transforms can be developed that will run on Apple silicon Macs.
Desktops such as Docker Desktop, Podman desktop and Rancher desktop use different virtualization and emulation technics, (qemu, Apple Virtualization framework) to allow the execution of containers based on images compiled for Intel silicon. However, emulation significantly impacts performance, and there are additional restrictions, such as Virtual Machine RAM size.
On the other hand, executing a Kind Kubernetes cluster with KubeFlow pipelines (KFP) and local data storage (Minio) requires a significant amount of memory. For this initial Data Prep Kit release, we do not recommend local (Kind) execution on Mac computers with Apple silicon. Instead, we suggest using a real Kubernetes cluster or a Linux virtual machine with an Intel CPU.
To verify that running transforms through KFP does not leak memory and also get an idea on the required Podman VM memory size configuration, a few tests were devised and run, as summarized here.
Note: the current release does not support building cross-platform images, therefore, please do not build images on the Apple silicon.