az ml 2.24.0 not compatible with mcr.microsoft.com/azure-cli:2.58.0

[danigian]

Describe the bug

az ml 2.24.0 extension is not compatible with container image 'mcr.microsoft.com/azure-cli:2.58.0'

This is related to the newly added dependency on azureml-dataprep-rslex which has no musllinux compatible distributions

Related command

az extension add --name ml --version 2.24.0

Errors

az extension add --name ml --version 2.24.0

Default enabled including preview versions for extension installation now. Disabled in May 2024. Use '--allow-preview true' to enable it specifically if needed. Use '--allow-preview false' to install stable version only.
An error occurred. Pip failed with status code 1. Use --debug for more information.

Issue script & Debug output

...
INFO: pip is looking at multiple versions of ml to determine which version is compatible with other requirements. This could take a while.
Will try a different candidate, due to conflict:
    ml 2.24.0 depends on azureml-dataprep-rslex>=2.22.0
ERROR: Could not find a version that satisfies the requirement azureml-dataprep-rslex>=2.22.0 (from ml) (from versions: none)
ERROR: No matching distribution found for azureml-dataprep-rslex>=2.22.0
...

Expected behavior

I should be able to install the ML extension on a Microsoft-provided Azure CLI container image

Environment Summary

azure-cli 2.58.0

core 2.58.0
telemetry 1.1.0

Dependencies:
msal 1.26.0
azure-mgmt-resource 23.1.0b2

Python location '/usr/local/bin/python'
Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.11.8 (main, Feb 7 2024, 23:20:20) [GCC 13.2.1 20231014]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[jsburckhardt]

same here - I reproduced it within the container and try install ml extension. Meanwhile if you install v2.23.0 it works but it was complaining about some packages:

• If i try directly 2.23.0:

docker run -it --rm  mcr.microsoft.com/azure-cli:2.58.0 az extension add -n ml --version 2.23.0 --debug

error:

• Ifi i install the packages in the list it works

apk add gcc python3-dev musl-dev linux-headers
az extension add --name ml --version  2.23.0

[YusakuNo1]

Summary:

1. CLI 2.24.0 introduces a hard dependency to package azureml-dataprep-rslex. azureml-dataprep-rslex was the dependency for ML extension in 3+ years, but it was optional.
2. azureml-dataprep-rslex doesn't have a release for Linux Alpine. azureml-dataprep-rslex team claimed to support Linux Alpine could be hard and they don't have requests to support Linux Alpine distribution.

Options (details are in internal thread):

• Remove the hard dependency of azureml-dataprep-rslex from Azure CLI ML extension.
• Skip the metadata checking for this dependency.

Next Step:
The PR for option 1 is ready, while we are still looking for the input from Azure CLI team for option 2

[YusakuNo1]

FYI: the fix is released at CLI 2.25.1, please verify