mcr.microsoft.com/azureml/curated/foundation-model-inference:31 - unable to deploy MLFlow Model -llama2-70b

[jakeatmsft]

Steps to reproduce

When attempting to deploy a packaged model of Llama2-70b to the mcr.microsoft.com/azureml/curated/foundation-model-inference:31 image the deployment fails when attempting to start the api_server.py.

Upon debugging, pip install of the requirements.txt shows error in pandas deployment due to python version mismatch:

root@ac4880151333:/# pip install -r requirements.txt
Collecting deepspeed==0.12.5
Downloading deepspeed-0.12.5.tar.gz (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 14.9 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting deepspeed-mii==0.1.3
Downloading deepspeed_mii-0.1.3-py3-none-any.whl (108 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 108.1/108.1 kB 15.5 MB/s eta 0:00:00
Collecting vllm==0.2.7
Downloading vllm-0.2.7-cp38-cp38-manylinux1_x86_64.whl (10.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.2/10.2 MB 82.1 MB/s eta 0:00:00
Collecting deepspeed-kernels==0.0.1.dev1698255861
Downloading deepspeed_kernels-0.0.1.dev1698255861-py3-none-manylinux1_x86_64.whl (44.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.5/44.5 MB 27.6 MB/s eta 0:00:00
Collecting diffusers==0.23.0
Downloading diffusers-0.23.0-py3-none-any.whl (1.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 64.9 MB/s eta 0:00:00
ERROR: Ignored the following versions that require a different python version: 2.1.0 Requires-Python >=3.9; 2.1.0rc0 Requires-Python >=3.9; 2.1.1 Requires-Python >=3.9; 2.1.2 Requires-Python >=3.9; 2.1.3 Requires-Python >=3.9; 2.1.4 Requires-Python >=3.9; 2.2.0 Requires-Python >=3.9; 2.2.0rc0 Requires-Python >=3.9; 2.2.1 Requires-Python >=3.9; 2.2.2 Requires-Python >=3.9
ERROR: Could not find a version that satisfies the requirement pandas~=2.1.4 (from versions: 0.1, 0.2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.21.1, 0.22.0, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0, 0.24.1, 0.24.2, 0.25.0, 0.25.1, 0.25.2, 0.25.3, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.4.0rc0, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.5.0rc0, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 2.0.0rc0, 2.0.0rc1, 2.0.0, 2.0.1, 2.0.2, 2.0.3)
ERROR: No matching distribution found for pandas~=2.1.4

requirements.txt:
root@ac4880151333:/# cat requirements.txt
deepspeed==0.12.5
deepspeed-mii==0.1.3
vllm==0.2.7
deepspeed-kernels==0.0.1.dev1698255861
diffusers==0.23.0
pandas~=2.1.4
transformers~=4.38.0
aiolimiter~=1.1.0
azure-ai-contentsafety==1.0.0b1
azure-ai-ml==1.13.0
azure-identity==1.15.0
azureml-ai-monitoring==0.1.0b4
pydantic conflict resolution
azureml-inference-server-http==0.7.7
azureml-mlflow==1.55.0
cryptography~=42.0.4
certifi==2023.11.17
requests~=2.31.0
aiohttp~=3.9.1
torch~=2.1.2

Expected behavior

MLFlow type models should deploy correctly when using pre-defined docker environment and scoring files.

Actual behavior

Model deployment to managed online endpoint fails due to a failure in the docker file:

Error

root@ac4880151333:/# pip install -r requirements.txt
Collecting deepspeed==0.12.5
Downloading deepspeed-0.12.5.tar.gz (1.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 14.9 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting deepspeed-mii==0.1.3
Downloading deepspeed_mii-0.1.3-py3-none-any.whl (108 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 108.1/108.1 kB 15.5 MB/s eta 0:00:00
Collecting vllm==0.2.7
Downloading vllm-0.2.7-cp38-cp38-manylinux1_x86_64.whl (10.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.2/10.2 MB 82.1 MB/s eta 0:00:00
Collecting deepspeed-kernels==0.0.1.dev1698255861
Downloading deepspeed_kernels-0.0.1.dev1698255861-py3-none-manylinux1_x86_64.whl (44.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.5/44.5 MB 27.6 MB/s eta 0:00:00
Collecting diffusers==0.23.0
Downloading diffusers-0.23.0-py3-none-any.whl (1.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 64.9 MB/s eta 0:00:00
ERROR: Ignored the following versions that require a different python version: 2.1.0 Requires-Python >=3.9; 2.1.0rc0 Requires-Python >=3.9; 2.1.1 Requires-Python >=3.9; 2.1.2 Requires-Python >=3.9; 2.1.3 Requires-Python >=3.9; 2.1.4 Requires-Python >=3.9; 2.2.0 Requires-Python >=3.9; 2.2.0rc0 Requires-Python >=3.9; 2.2.1 Requires-Python >=3.9; 2.2.2 Requires-Python >=3.9
ERROR: Could not find a version that satisfies the requirement pandas~=2.1.4 (from versions: 0.1, 0.2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.21.1, 0.22.0, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0, 0.24.1, 0.24.2, 0.25.0, 0.25.1, 0.25.2, 0.25.3, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.4.0rc0, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.5.0rc0, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 2.0.0rc0, 2.0.0rc1, 2.0.0, 2.0.1, 2.0.2, 2.0.3)
ERROR: No matching distribution found for pandas~=2.1.4

requirements.txt:

root@ac4880151333:/# cat requirements.txt
deepspeed==0.12.5
deepspeed-mii==0.1.3
vllm==0.2.7
deepspeed-kernels==0.0.1.dev1698255861
diffusers==0.23.0
pandas~=2.1.4
transformers~=4.38.0
aiolimiter~=1.1.0
azure-ai-contentsafety==1.0.0b1
azure-ai-ml==1.13.0
azure-identity==1.15.0
azureml-ai-monitoring==0.1.0b4
pydantic conflict resolution
azureml-inference-server-http==0.7.7
azureml-mlflow==1.55.0
cryptography~=42.0.4
certifi==2023.11.17
requests~=2.31.0
aiohttp~=3.9.1
torch~=2.1.2

Addition information

No response

[vizhur]

foundation-model-inference is on py310 since version 19, that should satisfy pandas ~=2.1.4 interpreter requirements. Can you please confirm you are on a recent version of the environment/image and also provide corresponding output from the container?