You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It seems that deepspeed doesn't use cuda installed in virtual environment.
To Reproduce
Steps to reproduce the behavior:
Remove your nvcc installed globally.
Create new virtual environment and activate it.
Install deepspeed==0.14.2
Write codes below and run it.
source code
importdeepspeed
error log
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/.venv/lib/python3.9/site-packages/deepspeed/__init__.py", line 25, in <module>
from . import ops
File "/.venv/lib/python3.9/site-packages/deepspeed/ops/__init__.py", line 15, in <module>
from ..git_version_info import compatible_ops as __compatible_ops__
File "/.venv/lib/python3.9/site-packages/deepspeed/git_version_info.py", line 29, in <module>
op_compatible = builder.is_compatible()
File "/.venv/lib/python3.9/site-packages/deepspeed/ops/op_builder/fp_quantizer.py", line 29, in is_compatible
sys_cuda_major, _ = installed_cuda_version()
File "/.venv/lib/python3.9/site-packages/deepspeed/ops/op_builder/builder.py", line 50, in installed_cuda_version
raise MissingCUDAException("CUDA_HOME does not exist, unable to compile CUDA op(s)")
deepspeed.ops.op_builder.builder.MissingCUDAException: CUDA_HOME does not exist, unable to compile CUDA op(s)
But cuda is located in virtual environment.
ls .venv/lib/python3.9/site-packages/nvidia | grep cuda
cuda_cupti
cuda_nvrtc
cuda_runtime
Expected behavior
It might be better if deepspeed uses cuda installed in virtual environment.
Assume that cuda runtime is installed globally 11.6 and torch uses cuda runtime 12.1. In this case deepspeed and pytorch don't use same version of cuda. It can cause confusion.
deepspeed requires cuda installed globally and torch install cuda in virtual environment. It increases docker image size.
nvcc might be unnecessary to check cuda version. nvcc is a compiler. To minimize docker images I don't want to install nvcc. But deepspeed uses it to check cuda version. We can use torch and print torch.cuda.version without nvcc.
System info (please complete the following information):
OS: centos 7
GPU count and types single machine with x8 V100s each]
Python 3.9.17
The text was updated successfully, but these errors were encountered:
Hi @zsaladin - the cuda runtime version and the cuda version that is bundled with torch can be different, so that is the reason why we need to be able to check the actual cuda runtime version that is installed.
@loadams Thanks for replying. I have some questions about your answer.
As you mentioned versions of cuda runtime installed globally and bundled with torch can be different. The cuda runtime bundled with torch is in virtual environment.
Does deepspeed use cuda runtime in virtual environment? If deepspeed uses cuda runtime in virtual environment then the version conflict cannot happen. So it would be great that deepspeed will use bundled cuda runtime if deepspeed doesn't use it now.
When I use deepspeed==0.12.6 it doesn't require nvcc. nvcc is a compiler not for checking version.
I'm not sure that but deepspeed needs to complie cuda code?
Hi @zsaladin - DeepSpeed uses the version of cuda runtime that is installed on the system, it cannot "use" the version that torch is built with, as that doesn't have nvcc/cuda drivers, it is just what the installed pytorch is built against.
As for why it didn't require nvcc in 0.12.6, we will have to check the code to see what changes have taken place that would cause this. With 0.12.6 does DeepSpeed detect that you are using an Nvidia GPU? Are you able to run nvidia-smi on your system with DeepSpeed 0.14.x?
Describe the bug
It seems that deepspeed doesn't use cuda installed in virtual environment.
To Reproduce
Steps to reproduce the behavior:
Remove your nvcc installed globally.
Create new virtual environment and activate it.
Install
deepspeed==0.14.2
Write codes below and run it.
source code
error log
But cuda is located in virtual environment.
ls .venv/lib/python3.9/site-packages/nvidia | grep cuda cuda_cupti cuda_nvrtc cuda_runtime
Expected behavior
It might be better if deepspeed uses cuda installed in virtual environment.
Assume that cuda runtime is installed globally 11.6 and torch uses cuda runtime 12.1. In this case deepspeed and pytorch don't use same version of cuda. It can cause confusion.
deepspeed requires cuda installed globally and torch install cuda in virtual environment. It increases docker image size.
nvcc
might be unnecessary to check cuda version.nvcc
is a compiler. To minimize docker images I don't want to installnvcc
. But deepspeed uses it to check cuda version. We can use torch and printtorch.cuda.version
withoutnvcc
.System info (please complete the following information):
The text was updated successfully, but these errors were encountered: