-
Notifications
You must be signed in to change notification settings - Fork 21.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CUDA memory leak? #1230
Comments
Please try running with cudnn.benchmark=False. cudnn.benchmark is not doing any good for dilated convolutions anyway. |
Hi,
Also the |
Wow, thank you for the quick and very precise answers! @ngimel: Never thought of that... Using cudnn.benchmark=False did the trick. No matter what size of the list I tried. The error did not show again. @albanD: That did it, as well! I did not use set_device() because its doc states its discouraged in favor of device(). Strange. |
* Add triton build scripts for docker build * Match permissions to upstream * Update triton commit to tip of ROCm triton's release/pytorch_2.0 branch * typo
Output:
I was wondering why I got the CUDNN_STATUS_ALLOC_FAILED.
After some experiments I found out that - error or not - depends on the sequence in the dilation list:
line 37:
for dilation in reversed([64, 128, 256, 512]):
Execution without
reversed
goes without error.I am not yet familiar with the whole thing. I am I missing something?
--
I thankfully adapted this code from #967.
By the way: I am also curious why I can’t change the active CUDA device (see comment in the code)… but I probably just need to get more into it…
The text was updated successfully, but these errors were encountered: