Is it possible to use python cuda libraries from a virtual env? Origianal whisper is super easy to install since it doesn't require me to change my system cuda version and simply pulls in the needed libs using pip. #153

Gobz · 2023-04-15T17:18:00Z

No description provided.

guillaumekln · 2023-04-16T09:26:00Z

Currently you can already make use of these libraries but you need to manually set the environment variable LD_LIBRARY_PATH before running Python. I verified that the following works:

pip install nvidia-cublas-cu11 nvidia-cudnn-cu11

export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`

(Note that this only works on Linux systems.)

I will look if we can load these libraries automatically when they are installed. It's an improvement we should make in the underlying CTranslate2 library.

hoonlight · 2023-06-10T15:01:53Z

Currently you can already make use of these libraries but you need to manually set the environment variable LD_LIBRARY_PATH before running Python. I verified that the following works:
pip install nvidia-cublas-cu11 nvidia-cudnn-cu11

export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
(Note that this only works on Linux systems.)

I will look if we can load these libraries automatically when they are installed. It's an improvement we should make in the underlying CTranslate2 library.

Is it possible to get it to work with VENV on Windows by any chance? I have cuda 11.8, cudnn 11.x all installed properly, but it's not working.

Could not load library cudnn_cnn_infer64_8.dll. Error code 126
Please make sure cudnn_cnn_infer64_8.dll is in your library path!

would installing pytorch help?

What's really weird is that a few days ago, it worked fine without any CUDA, CUDNN, zlib installation. After a clean install of windows, it doesn't work.

edited: The same error occurs even in a non-virtual environment. I did everything, including cuda 11.8 installation, cudnn 11.x installation, zlib installation, and path addition, but I don't know why this is happening.

guillaumekln · 2023-06-12T08:02:12Z

This solution does not work on Windows because NVIDIA only provides Linux binaries for the cuDNN package:

https://pypi.org/project/nvidia-cudnn-cu11/#files

Installing PyTorch will not help in this case.

I did everything, including cuda 11.8 installation, cudnn 11.x installation, zlib installation, and path addition, but I don't know why this is happening.

Maybe you should double-check the PATH setting. I know it can be tricky to get it right.

You could also look at @Purfview's standalone executable: https://github.com/Purfview/whisper-standalone-win. There you can download the NVIDIA libraries and simply put them in the same directory as the executable.

hoonlight · 2023-06-12T13:04:16Z

This solution does not work on Windows because NVIDIA only provides Linux binaries for the cuDNN package:

https://pypi.org/project/nvidia-cudnn-cu11/#files

Installing PyTorch will not help in this case.

I did everything, including cuda 11.8 installation, cudnn 11.x installation, zlib installation, and path addition, but I don't know why this is happening.

Maybe you should double-check the PATH setting. I know it can be tricky to get it right.

You could also look at @Purfview's standalone executable: https://github.com/Purfview/whisper-standalone-win. There you can download the NVIDIA libraries and simply put them in the same directory as the executable.

Thanks, I'll try again to make sure I'm not missing anything.
I just have one question: is the installation of cuda, cudnn "essential" for running faster-whisper on a gpu?
I don't remember installing cudnn (before the windows reinstall), but only because device="cuda" just worked fine.
If they are required to be installed, my memory is probably wrong.

guillaumekln · 2023-06-12T13:06:37Z

Yes these libraries are required for GPU execution. An error is raised when you try to use the GPU but these libraries cannot be found.

yangyaofei · 2023-06-13T08:22:20Z

According to this solution, I get the code below:

try:
    import os
    import nvidia.cublas.lib
    import nvidia.cudnn.lib

    cublas_path = os.path.dirname(nvidia.cublas.lib.__file__)
    cudnn_path = os.path.dirname(nvidia.cudnn.lib.__file__)
    os.environ["LD_LIBRARY_PATH"] = f"{cublas_path}:{cudnn_path}"
except ModuleNotFoundError:
    pass

But still get same error, like the PATH is not set.

guillaumekln · 2023-06-13T08:26:05Z

The environment variable LD_LIBRARY_PATH should be set before starting the Python process.

yangyaofei · 2023-06-13T08:30:17Z

@guillaumekln oh, thank you to explain this. It seems like the once for all solution is not easy😂.

Feanix-Fyre · 2023-06-13T21:37:16Z

I'm pretty certain that NVIDIA offers cuDNN files of some sort for Windows at https://developer.nvidia.com/rdp/cudnn-download. I do remember getting faster-whisper to work on the GPU on Windows but I do remember it being quite the hassle, either due to my inexperience or genuine difficulty.

guillaumekln · 2023-06-13T21:42:17Z

Yes, you can download cuDNN binaries for Windows on the NVIDIA website.

But this issue is about installing cuDNN via PyPI with:

pip install nvidia-cudnn-cu11

This does not work on Windows.

lugia19 · 2023-07-03T13:28:46Z

If you're on Windows, a functioning workaround I've found is to install torch with cuda support, then add the "lib" subfolder to your PATH.

It works since the lib folder contains DLLs for both cuBLAS and cuDNN.

vadi2 · 2023-08-01T16:16:19Z

The trick in #153 (comment) is not working for me, what am I doing wrong?

(faster-whisper) vadi@barbar:~$ pip install nvidia-cublas-cu11 nvidia-cudnn-cu11
Requirement already satisfied: nvidia-cublas-cu11 in ./Programs/miniconda3/envs/faster-whisper/lib/python3.10/site-packages (11.11.3.6)
Requirement already satisfied: nvidia-cudnn-cu11 in ./Programs/miniconda3/envs/faster-whisper/lib/python3.10/site-packages (8.9.2.26)
(faster-whisper) vadi@barbar:~$ export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
(faster-whisper) vadi@barbar:~$ echo $LD_LIBRARY_PATH 
/home/vadi/Programs/miniconda3/envs/faster-whisper/lib/python3.10/site-packages/nvidia/cublas/lib:/home/vadi/Programs/miniconda3/envs/faster-whisper/lib/python3.10/site-packages/nvidia/cudnn/lib
(faster-whisper) vadi@barbar:~$ python ~/Downloads/faster-whisper.py 
Traceback (most recent call last):
  File "/home/vadi/Downloads/faster-whisper.py", line 6, in <module>
    model = WhisperModel(model_size, device="cuda", compute_type="float16")
  File "/home/vadi/Programs/miniconda3/envs/faster-whisper/lib/python3.10/site-packages/faster_whisper/transcribe.py", line 124, in __init__
    self.model = ctranslate2.models.Whisper(
RuntimeError: CUDA failed with error unknown error
(faster-whisper) vadi@barbar:~$

Running Nvidia driver 535.54.03 on RTX 4080 on Ubuntu 22.04.

guillaumekln · 2023-08-02T05:55:44Z

This is probably another issue. I think it happens when the GPU driver is not loaded correctly (e.g. it was just updated to a new version). Rebooting the system will often fix this type of error.

vadi2 · 2023-08-02T06:35:32Z

It did. Thanks!

Simon-chai · 2023-09-20T10:37:42Z

nvidia.cublas.lib.__file__

why my nvidia.cublas.lib.file attribute is None so the environment variable failed to set,and when i run faster_whisper and will encounter the error
"Could not load library libcudnn_ops_infer.so.8. Error: libcudnn_ops_infer.so.8: cannot open shared object file: No such file or directory
Please make sure libcudnn_ops_infer.so.8 is in your library path!"

guillaumekln · 2023-09-21T11:49:36Z

The error is about cuDNN, not cuBLAS.

You should double-check that you correct installed the pip packages as shown in #153 (comment).

s-h-a-d-o-w · 2023-10-06T18:32:29Z

Currently you can already make use of these libraries but you need to manually set the environment variable LD_LIBRARY_PATH before running Python. I verified that the following works:
pip install nvidia-cublas-cu11 nvidia-cudnn-cu11

export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
(Note that this only works on Linux systems.)

I will look if we can load these libraries automatically when they are installed. It's an improvement we should make in the underlying CTranslate2 library.

Doesn't work for me on Ubuntu (WSL).

__file__ doesn't exist:

>>> print(nvidia.cublas.lib.__file__)
None

But the libraries are installed:

andy@work:~$ pip install nvidia-cublas-cu11 nvidia-cudnn-cu11
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: nvidia-cublas-cu11 in ./.local/lib/python3.8/site-packages (11.10.3.66)
Requirement already satisfied: nvidia-cudnn-cu11 in ./.local/lib/python3.8/site-packages (8.5.0.96)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from nvidia-cublas-cu11) (68.2.2)
Requirement already satisfied: wheel in ./.local/lib/python3.8/site-packages (from nvidia-cublas-cu11) (0.40.0)

>>> import os; import nvidia.cublas.lib; import nvidia.cudnn.lib;
>>> print(nvidia.cublas.lib)
<module 'nvidia.cublas.lib' (namespace)>

Setting the paths manually like so worked:

export LD_LIBRARY_PATH="$HOME/.local/lib/python3.8/site-packages/nvidia/cublas/lib/:$HOME/.local/lib/python3.8/site-packages/nvidia/cudnn/lib/"

gu-ma · 2023-11-30T20:50:41Z

Currently you can already make use of these libraries but you need to manually set the environment variable LD_LIBRARY_PATH before running Python. I verified that the following works:
pip install nvidia-cublas-cu11 nvidia-cudnn-cu11

export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
(Note that this only works on Linux systems.)

I will look if we can load these libraries automatically when they are installed. It's an improvement we should make in the underlying CTranslate2 library.

Very cool, thanks for the tip! Btw if you're using a conda env you can set the env var like this (in your environment):

conda env config vars set LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`

It will overwrite the default path

pauloboritza · 2023-12-13T01:47:21Z

This solution does not work on Windows because NVIDIA only provides Linux binaries for the cuDNN package:
https://pypi.org/project/nvidia-cudnn-cu11/#files
Installing PyTorch will not help in this case.

I did everything, including cuda 11.8 installation, cudnn 11.x installation, zlib installation, and path addition, but I don't know why this is happening.

Maybe you should double-check the PATH setting. I know it can be tricky to get it right.
You could also look at @Purfview's standalone executable: https://github.com/Purfview/whisper-standalone-win. There you can download the NVIDIA libraries and simply put them in the same directory as the executable.

Thanks, I'll try again to make sure I'm not missing anything. I just have one question: is the installation of cuda, cudnn "essential" for running faster-whisper on a gpu? I don't remember installing cudnn (before the windows reinstall), but only because device="cuda" just worked fine. If they are required to be installed, my memory is probably wrong.

Error in Windows

....
RuntimeError: Library cublas64_11.dll is not found or cannot be loaded

download https://github.com/Purfview/whisper-standalone-win/releases/download/libs/cuBLAS.and.cuDNN_win_v3.zip
unizip *.dll in project path
add lines in the app:

import ctypes
cublas64_11 = ctypes.WinDLL('.\cublas64_11.dll')

santiago-afonso · 2024-02-07T12:04:15Z

If you're on Windows, a functioning workaround I've found is to install torch with cuda support, then add the "lib" subfolder to your PATH.

It works since the lib folder contains DLLs for both cuBLAS and cuDNN.

It's been a while, but what command did you use to install torch specifically?

lugia19 · 2024-02-07T12:06:14Z

If you're on Windows, a functioning workaround I've found is to install torch with cuda support, then add the "lib" subfolder to your PATH.
It works since the lib folder contains DLLs for both cuBLAS and cuDNN.

It's been a while, but what command did you use to install torch specifically?

Oh, it was just the usual pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 type deal.

Just installing torch with CUDA.

louistiti · 2024-05-21T15:13:54Z

Currently you can already make use of these libraries but you need to manually set the environment variable LD_LIBRARY_PATH before running Python. I verified that the following works:
pip install nvidia-cublas-cu11 nvidia-cudnn-cu11

export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
(Note that this only works on Linux systems.)

I will look if we can load these libraries automatically when they are installed. It's an improvement we should make in the underlying CTranslate2 library.

Hi @guillaumekln , may I know if this improvement has been made on CTranslate2? I'm using CUDA 12.1 and I'm facing the same issue. It works by exporting the LD_LIBRARY_PATH env variable though.

Happy to help for anything you'd need.

guillaumekln pinned this issue May 4, 2023

guillaumekln mentioned this issue Jun 15, 2023

Please make sure libcudnn_ops_infer.so.8 is in your library path! #297

Closed

guillaumekln mentioned this issue Jul 25, 2023

Kernel crash with Tesla P40 GPU on CUDA 12.1, but works fine on Google Colab with CUDA 12.0 #378

Closed

edurenye mentioned this issue Oct 6, 2023

[WIP] Add initial GPU support rhasspy/wyoming-addons#4

Open

santiago-afonso mentioned this issue Feb 7, 2024

CUDA requirements need to match project dependencies. Distribute the libraries? savbell/whisper-writer#33

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Is it possible to use python cuda libraries from a virtual env? Origianal whisper is super easy to install since it doesn't require me to change my system cuda version and simply pulls in the needed libs using pip. #153

Is it possible to use python cuda libraries from a virtual env? Origianal whisper is super easy to install since it doesn't require me to change my system cuda version and simply pulls in the needed libs using pip. #153

Gobz commented Apr 15, 2023 •

edited

guillaumekln commented Apr 16, 2023

hoonlight commented Jun 10, 2023 •

edited

guillaumekln commented Jun 12, 2023

hoonlight commented Jun 12, 2023 •

edited

guillaumekln commented Jun 12, 2023

yangyaofei commented Jun 13, 2023

guillaumekln commented Jun 13, 2023

yangyaofei commented Jun 13, 2023

Feanix-Fyre commented Jun 13, 2023

guillaumekln commented Jun 13, 2023

lugia19 commented Jul 3, 2023

vadi2 commented Aug 1, 2023

guillaumekln commented Aug 2, 2023

vadi2 commented Aug 2, 2023

Simon-chai commented Sep 20, 2023

guillaumekln commented Sep 21, 2023

s-h-a-d-o-w commented Oct 6, 2023

gu-ma commented Nov 30, 2023

pauloboritza commented Dec 13, 2023

santiago-afonso commented Feb 7, 2024

lugia19 commented Feb 7, 2024

louistiti commented May 21, 2024

Is it possible to use python cuda libraries from a virtual env? Origianal whisper is super easy to install since it doesn't require me to change my system cuda version and simply pulls in the needed libs using pip. #153

Is it possible to use python cuda libraries from a virtual env? Origianal whisper is super easy to install since it doesn't require me to change my system cuda version and simply pulls in the needed libs using pip. #153

Comments

Gobz commented Apr 15, 2023 • edited

guillaumekln commented Apr 16, 2023

hoonlight commented Jun 10, 2023 • edited

guillaumekln commented Jun 12, 2023

hoonlight commented Jun 12, 2023 • edited

guillaumekln commented Jun 12, 2023

yangyaofei commented Jun 13, 2023

guillaumekln commented Jun 13, 2023

yangyaofei commented Jun 13, 2023

Feanix-Fyre commented Jun 13, 2023

guillaumekln commented Jun 13, 2023

lugia19 commented Jul 3, 2023

vadi2 commented Aug 1, 2023

guillaumekln commented Aug 2, 2023

vadi2 commented Aug 2, 2023

Simon-chai commented Sep 20, 2023

guillaumekln commented Sep 21, 2023

s-h-a-d-o-w commented Oct 6, 2023

gu-ma commented Nov 30, 2023

pauloboritza commented Dec 13, 2023

santiago-afonso commented Feb 7, 2024

lugia19 commented Feb 7, 2024

louistiti commented May 21, 2024

Gobz commented Apr 15, 2023 •

edited

hoonlight commented Jun 10, 2023 •

edited

hoonlight commented Jun 12, 2023 •

edited