UNAVAILABLE: Internal: FileNotFoundError: [Errno2] No usable temporary directory found in ['/tp', '/var/tmp','/usr/tmp', '/tmp/python_Lsk/3'] env_6Lp

[VirginieBfd]

Description

Unable to load models with an attached conda env:

UNAVAILABLE: Internal: FileNotFoundError: [Errno2] No usable temporary directory found in ['/tp', '/var/tmp','/usr/tmp', '/tmp/python_Lsk/3'] env_6Lp

My config looks like

name: "model_a"
backend: "python"
...
parameters: {
  key: "EXECUTION_ENV_PATH",
  value: {string_value: "$$TRITON_MODEL_DIRECTORY/model_a.tar.gz"}
}

E0417 14:59:28.071079 1 backend_model.cc:635] ERROR: Failed to create instance: FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/tmp/python_env_6LpLsk/3']

How I created my env:

export PYTHONNOUSERSITE=True
conda create -n model_a python=3.10 -y
conda activate model_a
pip install -r requirements.txt
pip install conda-pack
conda install -c conda-forge libstdcxx-ng=12 -y
conda pack

Triton Information

I am using:

FROM 785573368785.dkr.ecr.us-east-1.amazonaws.com/sagemaker-tritonserver:23.12-py3

Expected behavior

I would expect my model to be able to load, using this env.

[Tabrizian]

@VirginieBfd Can you share the full logs? It appears that the error is coming from your Python model. Also, are you able to reproduce the same error using the NGC containers (i.e. using nvcr.io/nvidia/tritonserver:23.12-py3 instead of the sage maker image).