New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR #1469
Comments
Same error happens even when there is no convolutional layers in model.
|
@bobiblazeski, have you found any resolution to this issue? I'm currently blocked by this same error. |
@adwellj Nope I'm training on CPU until this is resolved. |
@bobiblazeski, I punted over to trying on Windows and finally just got this working. I had to drop down to tfjs-node-gpu version 0.3.2 due to node-gyp issues. However, once I finally got it to install, I later ran in to this same CuDNN issue! Fortunately, using CUDA 9.0 (needed for 0.3.2 compatibility) I got a better error message before the "This is probably because cuDNN failed to initialize..." message, stating that tfjs-node-gpu was built against CuDNN version 7.2. Once I downloaded that version, everything is working. I haven't went back to see if I could get it to work on the LINUX install, but I'm hoping that this could just be a CuDNN version incompatibility issue that you could experiment with. Luckily CuDNN doesn't have an install / uninstall process; it's simply copying the extracted files in to a dedicated directory that you include in your system path. I hope that helps give you some possible direction! |
As explained in #671 (comment) There is workaround by setting global variable |
have you solve to use GPU ? |
I am having this issue too, but it seems to resolve itself only when I restart my computer. This seems rather odd to me. I notice that the issue tends to happen after terminating my application(s) that utilize tfjs. EDIT: I tried adding |
this seems to be a duplicate of #671 , we will close this and track the issue at one place.Thank you |
To get help from the community, we encourage using Stack Overflow and the
tensorflow.js
tag.TensorFlow.js version
{ 'tfjs-core': '1.0.3',
'tfjs-data': '1.0.3',
'tfjs-layers': '1.0.3',
'tfjs-converter': '1.0.3',
tfjs: '1.0.3',
'tfjs-node': '1.0.2' }
Browser version
Running on node
Ubuntu 18.04
Describe the problem or feature request
I'm unable to use cudnn convolutional layers in my model on
tfjs-node-gpu
Possibly related due to issues with RTX series in this tensorflow workaround there is suggestion to use
config.gpu_options.allow_growth = True
Is there such option in tensorflow js?
Code to reproduce the bug / link to feature request
The text was updated successfully, but these errors were encountered: