Multi_GPU MODEL EXAMPLE : Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED

[GeorgeJub]

I used the attached file multi_gpu_test (text file format)
multi_gpu_test.txt
following multi-gpu example Tensorflow 2.3.0 from Tensoflow api documentation , to test it . I have the following system parameters:
Windows 10
2 RTX 2080 Ti GPUS
tensorflow version 2.3.0
CUDA Version 10.1 with CUDNN 7.6.5

The second file attached includes the log file from the command window prompt that shows the problem :
2020-09-24 18:27:41.164048: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2020-09-24 18:27:41.165324: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2020-09-24 18:27:41.167037: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2020-09-24 18:27:41.167141: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2020-09-24 18:27:41.167768: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2020-09-24 18:27:41.168299: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2020-09-24 18:27:41.465828: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-09-24 18:27:42.555004: E tensorflow/stream_executor/cuda/cuda_dnn.cc:328] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2020-09-24 18:27:42.555249: E tensorflow/stream_executor/cuda/cuda_dnn.cc:328] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2020-09-24 18:27:42.557047: E tensorflow/stream_executor/cuda/cuda_dnn.cc:328] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
2020-09-24 18:27:42.558144: E tensorflow/stream_executor/cuda/cuda_dnn.cc:328] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED
[I 18:28:46.435 NotebookApp] Starting buffering

I am not sure what the problem is and how to figure a solution - Thank you for your help,
The sample code attached was pulled directly from the Tensorflow documentation site to demonstrate the consept of the multi_gpu operation -
George Jubran

log_file_from_command_promt_jupyter_notebook.txt

[ravikyram]

@GeorgeJub

Can you try Limiting GPU memory growth from here.

Please, refer SO link, link1 and see if it helps you.

Also, refer tested build configurations from here.Thanks!

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.