Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Could not create cudnn handle #18

Closed
tabelh opened this issue Mar 28, 2019 · 4 comments
Closed

Could not create cudnn handle #18

tabelh opened this issue Mar 28, 2019 · 4 comments

Comments

@tabelh
Copy link

tabelh commented Mar 28, 2019

Hello I am trying to use this driver in combination with a deep learning object detection model. The model and the camera are communicating just fine, this is could test by using a virtual camera that I setup in nxView. However when i switched over to the actual camera I get the error that the cudnn handle could not be created. The config file that I am using was saved with the use of CUDA disabled.
Is it possible that this is activated by default when running the camera with ros? If so how would i change the default to no use CUDA?

Thanks in advance!

@saierd
Copy link
Member

saierd commented Mar 28, 2019

What exactly is the message and which ROS node prints it? We do not use cuDNN, so the error message can't be from the Ensenso node or the NxLib. The NxLib can use CUDA, but that is disabled by default. Even if it's enabled, it shouldn't be able to interfere with another process using cuDNN.

@tabelh
Copy link
Author

tabelh commented Mar 28, 2019

Thanks for the quick response!
The statement, as i can see it, orginates from the model. I won't provide the entire trace as I don't think it'll be usefull:

E tensorflow/stream_executor/cuda/cuda_dnn.cc:334] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

I was just curious if the camera driver used CUDA. From what i have read about the error code it mostly pretains to a memory issue. But if the driver does not make use of CUDA the problem won't be on the camera side i recon. Still find it strange that the model does seem to be operational when using the virtual camera. Do they output data in the same format? If I remember correctly they do, i think one of your vision experts once told me in a tutorial.

@saierd
Copy link
Member

saierd commented Mar 28, 2019

Yes, grabbing data from virtual or real cameras doesn't make any difference. They both look the same in the NxLib API and output the same data.

Even if the NxLib would use CUDA, this should not cause any problems in your other ROS node. They run in different processes and both of them can only communicate with the GPU through the Nvidia driver. In the case of the virtual camera, we actually use the GPU for rendering and that seems to work.

Did you try the solution from e.g. tensorflow/tensorflow#24496?

@tabelh
Copy link
Author

tabelh commented Mar 28, 2019

I have looked into it, however, the code that I am trying to utilize does not have the explicit session statements that are given in the examples so i am unsure where to add the line.
Thank you very much for the help but it seems I need to take my issue elsewhere. ;)

@tabelh tabelh closed this as completed Mar 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants