Facing GPU Configuration Issues in Running NNI Experiment

[Baiyun-u-smartAI]

Hello NNI Community,

I've been working on a neural architecture search using the NNI tutorial example. The code runs successfully without GPU configuration. However, when I try to modify it for GPU usage with exp.config.trial_gpu_number = 1 and exp.config.training_service.use_active_gpu = True, it fails.

Here are the error logs:
experiment.log

My conda environment is as follows:

aiohttp 3.9.1
aiosignal 1.3.1
archspec 0.2.1
astor 0.8.1
async-timeout 4.0.3
attrs 23.1.0
blas 1.0
boltons 23.0.0
brotli-python 1.0.9
bzip2 1.0.8
ca-certificates 2023.08.22
certifi 2023.11.17
cffi 1.15.1
charset-normalizer 2.0.4
cloudpickle 3.0.0
colorama 0.4.6
conda 23.11.0
conda-content-trust 0.2.0
conda-libmamba-solver 23.11.1
conda-package-handling 2.2.0
conda-package-streaming 0.9.0
console_shortcut_miniconda 0.1.1
contextlib2 21.6.0
contourpy 1.1.1
cryptography 41.0.3
cuda-cccl 12.3.101
cuda-cudart 11.8.89
cuda-cudart-dev 11.8.89
cuda-cupti 11.8.87
cuda-libraries 11.8.0
cuda-libraries-dev 11.8.0
cuda-nvrtc 11.8.89
cuda-nvrtc-dev 11.8.89
cuda-nvtx 11.8.86
cuda-profiler-api 12.3.101
cuda-runtime 11.8.0
cycler 0.12.1
distro 1.8.0
filelock 3.11.0
fmt 9.1.0
fonttools 4.46.0
freetype 2.12.1
frozenlist 1.4.0
fsspec 2023.12.1
giflib 5.2.1
gmpy2 2.1.2
idna 3.4
importlib-metadata 7.0.0
importlib-resources 6.1.1
intel-openmp 2023.1.0
jinja2 3.1.2
joblib 1.3.2
jpeg 9e
json-tricks 3.17.3
jsonpatch 1.32
jsonpointer 2.1
kiwisolver 1.4.5
lerc 3.0
libarchive 3.6.2
libcublas 11.11.3.6
libcublas-dev 11.11.3.6
libcufft 10.9.0.58
libcufft-dev 10.9.0.58
libcurand 10.3.4.101
libcurand-dev 10.3.4.101
libcurl 8.1.1
libcusolver 11.4.1.48
libcusolver-dev 11.4.1.48
libcusparse 11.7.5.86
libcusparse-dev 11.7.5.86
libdeflate 1.17
libffi 3.4.4
libiconv 1.16
libjpeg-turbo 2.0.0
libmamba 1.5.3
libmambapy 1.5.3
libnpp 11.8.0.86
libnpp-dev 11.8.0.86
libnvjpeg 11.9.0.86
libnvjpeg-dev 11.9.0.86
libpng 1.6.39
libsolv 0.7.24
libssh2 1.10.0
libtiff 4.5.1
libuv 1.44.2
libwebp 1.3.2
libwebp-base 1.3.2
libxml2 2.10.4
lightning-utilities 0.10.0
lz4-c 1.9.4
markupsafe 2.1.1
matplotlib 3.7.4
menuinst 1.4.19
mkl 2023.1.0
mkl-service 2.4.0
mkl_fft 1.3.8
mkl_random 1.2.4
mpc 1.1.0
mpfr 4.0.2
mpir 3.0.0
mpmath 1.3.0
multidict 6.0.4
networkx 3.1
nni 3.0
numpy 1.24.4
numpy-base 1.24.3
nvidia-ml-py 12.535.133
opencv-contrib-python 4.8.1.78
opencv-python 4.8.1.78
openjpeg 2.4.0
openssl 3.0.12
packaging 23.1
pandas 2.0.3
pcre2 10.42
pillow 10.0.1
pip 23.3
platformdirs 3.10.0
pluggy 1.0.0
powershell_shortcut_miniconda 0.0.1
prettytable 3.9.0
psutil 5.9.6
pybind11-abi 4
pycosat 0.6.6
pycparser 2.21
pyopenssl 23.2.0
pyparsing 3.1.1
pyqt5 5.15.10
pyqt5-qt5 5.15.2
pyqt5-sip 12.13.0
pyside2 5.15.2.1
pysocks 1.7.1
python 3.8.18
python-dateutil 2.8.2
pythonwebhdfs 0.2.3
pytorch 2.1.1
pytorch-cuda 11.8
pytorch-lightning 2.1.2
pytorch-mutex 1.0
pytz 2023.3.post1
pyyaml 6.0.1
reproc 14.2.4
reproc-cpp 14.2.4
requests 2.31.0
responses 0.24.1
ruamel.yaml 0.17.21
ruamel.yaml.clib 0.2.6
schema 0.7.5
scikit-learn 1.3.2
scipy 1.10.1
setuptools 68.0.0
shiboken2 5.15.2.1
simplejson 3.19.2
six 1.16.0
sqlite 3.41.2
sympy 1.12
tbb 2021.8.0
threadpoolctl 3.2.0
tk 8.6.12
torchaudio 2.1.1
torchmetrics 1.2.1
torchvision 0.16.1
tqdm 4.65.0
typeguard 4.1.2
typing_extensions 4.7.1
tzdata 2023.3
urllib3 1.26.18
vc 14.2
vs2015_runtime 14.27.29016
wcwidth 0.2.12
websockets 12.0
wheel 0.41.2
win_inet_pton 1.1.0
xz 5.4.2
yaml 0.2.5
yaml-cpp 0.8.0
yarl 1.9.4
zipp 3.17.0
zlib 1.2.13
zstandard 0.19.0
zstd 1.5.5

I've tried troubleshooting but to no avail. I would greatly appreciate any insights or suggestions from the community. If anyone has faced similar issues or knows a fix, please share your experience. Thank you very much for your help!

Best regards,
zlc@baiyunu.edu.cn