Ray error during noop pipeline run - local KFP and Kind

[ykoyfman]

Environment: KFP + Kind on Mac M1 Max with Docker and Colima. VM settings: default Running aarch64 8CPU 32GiB 1000GiB docker

During noop pipeline execution (all default values), execute ray step fails with:

[roytman]

tried to reproduce it on Mac M1 with Podman VM settings: 9 CPUs, 67 GB RAM

[roytman]

@Mohammad-nassar10 , can you try the test on your Intel-based laptop?

[Mohammad-nassar10]

I tried it with the default values that exist in the workflow and it passed and generated output file.

[roytman]

I have faced the issue when podman inspect or stop return "json: cannot unmarshal array into Go struct field InspectContainerConfig.Config.Entrypoint of type string. We have observed it with Hajar.
Therefore, I updated Podman to the latest version v5.0.2; it started successfully, but the Ray execution failed again with a different error

We decided we don't support Mac with Apple silicon, didn't we?

[blublinsky]

Works for me as well. Just re tested

[blublinsky]

Will try to work with Alexey later today to try it on his Mac

[roytman]

Restarted my laptop and tried it again with Podman.

It showed success, but we can see errors in the logs.

And actually, it submitted two jobs: one Failed, and another is Running for ever.

[blublinsky]

So I think the bottom line is that M1 with Podman is not a supported version for this release.

[blublinsky]

@roytman, @ykoyfman, @shahrokhDaijavad I think we can close this one

[shahrokhDaijavad]

@blublinsky I am ok with saying that a Mac M1 with Podman is not supported for kfp in this release, but of course it is ok for individual transforms and using S3-compatible Object Store locally. However, I want to make sure we get confirmation from @ykoyfman that he was able to run a full kfp test on a virtual X86 environment, before closing it.

[ykoyfman]

@shahrokhDaijavad - Agree - I'm continuing to test on X86 VM. Initially created a 32GB/8xCPU/100GB RHEL9 VM - Kubeflow could not completely start under Kind on this VM.

[blublinsky]

do not use RHEL9, use Ubuntu

[roytman]

Please do not close the issue, given that we have more time, I want to try to resolve the M1 limitations.

[shahrokhDaijavad]

Sure, Alexey.

[blublinsky]

@roytman, @ykoyfman, @shahrokhDaijavad. I think we tested this to death and have most of the answers. Can we, please, close it

[roytman]

no, as we discussed it still doesn't work on Mac M1. Somehow we should resolve it.

[shahrokhDaijavad]

@blublinsky: As you have seen in the tech slack channel, @ykoyfman is running this today on a RHEL VM. Once he is successful, we can close this, other than the Mac M1 problem that @roytman points to and may not have a short-term solution.

[shahrokhDaijavad]

Success by @ykoyfman today in running noop with kfp on RHEL VM! I close this issue for now, because I think Mac M1 may not be short-term.