ValueError: invalid literal for int() with base 10: ‘Failed to initialize NVML: Unknown Error’

[jldroid19]

🐛 Bug

q.app
q.user
q.client
report_error: True
q.events
q.args
report_error: True
stacktrace
Traceback (most recent call last):

File “/workspace/./llm_studio/app_utils/handlers.py”, line 78, in handle await home(q)

File “/workspace/./llm_studio/app_utils/sections/home.py”, line 66, in home stats.append(ui.stat(label=“Current GPU load”, value=f"{get_gpu_usage():.1f}%"))

File “/workspace/./llm_studio/app_utils/utils.py”, line 1949, in get_gpu_usage all_gpus = GPUtil.getGPUs()

File “/home/llmstudio/.local/share/virtualenvs/workspace-dqq3IVyd/lib/python3.10/site-packages/GPUtil/GPUtil.py”, line 102, in getGPUs deviceIds = int(vals[i])

ValueError: invalid literal for int() with base 10: ‘Failed to initialize NVML: Unknown Error’

Error
None

Git Version
fatal: not a git repository (or any of the parent directories): .git

To Reproduce

I'm not sure why this is happening. Hard to reproduce

LLM Studio version

v1.4.0-dev

[psinger]

This means you have no GPUs available. Can you run nvidia-smi to confirm everything is fine?

[jldroid19]

What's interesting is the environment just suddenly drops. It's like the GPU's just disappear after a few hours of training.

[psinger]

This seems to be an issue on your environment/system then unfortunately.

[psinger]

@jldroid19 did you figure the issue out?

[jldroid19]

@psinger I have not.

[psinger]

are you running this in docker?

[jldroid19]

are you running this in docker?

Yes I am running it using docker. It's strange due to the fact, we can run a dataset on it with an expected finish of 5 days and it'll finish. We then go to start another experiment and 3 hours later the container stops. Cause it to fail the experiment. With a quick docker restart the app is back up and running, but the training that had been going is lost.

[psinger]

I stumbled upon this recently, might be related:
NVIDIA/nvidia-docker#1469

NVIDIA/nvidia-container-toolkit#465 (comment)

There seems to be some issue of gpus being suddenly gone in Docker.