conda-pack failing: Failed to initialize Python stub for auto-complete

[jadhosn]

Description
All our attempts to create a conda environment, and leveraging it inside the triton serving container have failed. We're following the instructions for conda-pack on the Python backend snippet.

We're running a local test on a mac with no support to nvidia gpus. The eventual target is sagemaker instances.

This issue looks similar but didn't help.

Triton Information
What version of Triton are you using?

Python version=3.10.12
conda-pack=0.7.1 (conda-forge)
NVIDIA_TRITON_SERVER_VERSION=24.02
NVIDIA_TRITON_SERVER_BASE_VERSION=24.02
TRITON_SERVER_VERSION=2.43.0
TRITON_SERVER_GPU_ENABLED=1

Are you using the Triton container or did you build it yourself?
Using 24.02-pyt-python-py3 with additional environment variables for SageMaker endpoints

To Reproduce
Steps to reproduce the behavior.

tritonserver --log-verbose=true --allow-sagemaker=true --allow-grpc=false --allow-http=true --allow-metrics=true --model-control-mode=explicit --model-repository /opt/ml/models

#!/bin/bash

ENV_NAME=my_env

conda create -y --name $ENV_NAME python=3.10.12
conda activate $ENV_NAME
pip install pandas==1.5.3
pip install transformers==4.34.0
export PYTHONNOUSERSITE=True
conda install -y conda-pack=0.7.0
conda-pack

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).

This is a simple Python backend, running models.py that just imports the needed packages. For example:

import transformers

Expected behavior
A clear and concise description of what you expected to happen.

Creating a conda environment outside of our serving container, packaging it using conda-pack and leveraging it with Triton Serving Container.

Note: the combination of Python version, conda-pack and Triton server is not rigid, we can try any combination that works well together.

Traceback

Click me for a complete traceback

I0415 19:51:37.802987 194 python_be.cc:1942] Using Python execution env /opt/ml/models/ensemble_py_preprocess/env6.tar.gz
I0415 19:51:37.805128 194 pb_env.cc:292] Extracting Python execution env /opt/ml/models/ensemble_py_preprocess/env6.tar.gz
I0415 19:51:41.671836 194 stub_launcher.cc:388] Starting Python backend stub: source /tmp/python_env_BuKd6a/0/bin/activate && exec env LD_LIBRARY_PATH=/tmp/python_env_BuKd6a/0/lib:$LD_LIBRARY_PATH /opt/tritonserver/backends/python/triton_python_backend_stub /opt/ml/models/ensemble_py_preprocess/1/model.py triton_python_backend_shm_region_1 1048576 1048576 194 /opt/tritonserver/backends/python 336 ensemble_py_preprocess DEFAULT
I0415 19:51:42.521485 248 pb_stub.cc:290] Failed to initialize Python stub for auto-complete: ModuleNotFoundError: No module named '_multibytecodec'

At:
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/charset_normalizer/utils.py(10):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/charset_normalizer/md.py(10):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/charset_normalizer/cd.py(14):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/charset_normalizer/api.py(5):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/charset_normalizer/init.py(24):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/requests/compat.py(13):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/requests/exceptions.py(9):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/requests/init.py(45):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/huggingface_hub/hf_api.py(47):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load
(1050): _gcd_import
/tmp/python_env_BuKd6a/0/lib/python3.10/importlib/init.py(126): import_module
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/huggingface_hub/init.py(332): getattr
(1075): _handle_fromlist
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/transformers/utils/init.py(18):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load
(241): _call_with_frames_removed
(992): _find_and_load_unlocked
(1027): _find_and_load
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/transformers/dependency_versions_check.py(16):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load
(241): _call_with_frames_removed
(1086): _handle_fromlist
/tmp/python_env_BuKd6a/0/lib/python3.10/site-packages/transformers/init.py(26):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load
/opt/ml/models/ensemble_py_preprocess/1/model.py(1):
(241): _call_with_frames_removed
(883): exec_module
(703): _load_unlocked
(1006): _find_and_load_unlocked
(1027): _find_and_load

I0415 19:51:42.530535 194 python_be.cc:2360] TRITONBACKEND_ModelFinalize: delete model state

[VirginieBfd]

Hi, I am running into a similar issue. Did you find a fix?

[jadhosn]

Hi, I am running into a similar issue. Did you find a fix?

@VirginieBfd I haven't heard anything back from the Triton team. Are you also running on SageMaker endpoints? I believe that the error is driven by creating the environment on a different OS than the one running the Triton container. No solid leads yet.

[Tabrizian]

Thanks for filing a detailed issue. I have filed an internal ticket for the team to look at it.

[jadhosn]

@Tabrizian we've found the root cause.
As advised by conda-pack, the OS used to create an environment must match the hosting OS. In addition, these environments must be created within the a container that matches the hosting container. We installed miniconda on top of the NGC Triton container, created a conda environment and packed it from within the container.

See this reply on a recent Triton issue, it was what unblocked us.