Issues when using the '--cnn_scoring all' flag

[svolk19-stanford]

Issue summary

When I attempt to run a simple gnina command with the '--cnn_scoring all' flag, I receive the following output and error:

              _             
             ()            
   __ _ _ __  _ _ __   __ _ 
  / | '_ \| | '_ \ / _ |
 | (| | | | | | | | | (| |
  _, || |||| ||_,|
   / |                    
  |_/                     

gnina N/A N/A:N/A   Built Oct 31 2022.
gnina is based on smina and AutoDock Vina.
Please cite appropriately.

Commandline: gnina -r receptor.pdbqt -l ligand.pdbqt -o out.sdf --log log.txt --cnn_scoring all --config config.txt
Weights      Terms
-0.035579    gauss(o=0,_w=0.5,_c=8)
-0.005156    gauss(o=3,_w=2,_c=8)
0.840245     repulsion(o=0,_c=8)
-0.035069    hydrophobic(g=0.5,_b=1.5,_c=8)
-0.587439    non_dir_h_bond(g=-0.7,_b=0,_c=8)
1.923        num_tors_div

Detected 32 CPUs
WARNING: at low exhaustiveness, it may be impossible to utilize all CPUs
Reading input ... done.
Setting up the scoring function ... done.
Using random seed: 0
Performing search ... 
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
terminate called after throwing an instance of 'std::runtime_error'
  what():  CUDNN Error (out of memory?): CUDNN_STATUS_NOT_INITIALIZED
*** Aborted at 1696374248 (unix time) try "date -d @1696374248" if you are using GNU date ***
terminate called recursively
terminate called recursively
terminate called recursively
PC: @                0x0 (unknown)
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
*** SIGABRT (@0x5550600022fa3) received by PID 143267 (TID 0x7f6d1e7fc700) from PID 143267; stack trace: ***
    @     0x7f6e95b4eaae google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f6e83993630 (unknown)
    @     0x7f6e835ec387 __GI_raise
    @     0x7f6e835eda78 __GI_abort
    @     0x7f6e83e5134c _ZN9__gnu_cxx27__verbose_terminate_handlerEv.cold
    @     0x7f6e83e5c656 __cxxabiv1::__terminate()
    @     0x7f6e83e5c6c1 std::terminate()
    @     0x7f6e83e5c954 __cxa_throw
    @     0x7f6e97e2e718 caffe::CuDNNConvolutionLayer<>::LayerSetUp()
    @     0x7f6e97f60b4f caffe::Net<>::Init()
    @     0x7f6e97f6245e caffe::Net<>::Net()
    @           0x4d5775 CNNScorer::CNNScorer()
    @           0x53b5a2 parallel_mc_aux::operator()()
    @           0x537630 _ZN5boost6detail11thread_dataINS_17reference_wrapperIN12parallel_forIN13parallel_iterI15parallel_mc_auxNS_10ptr_vectorI16parallel_mc_taskNS_20heap_clone_allocatorEvEES7_ZNK11parallel_mcclERK5modelRNS6_I11output_typeS8_vEERK12precalculateR5igridRK3vecSO_RNS_6random23mersenne_twister_engineIjLm32ELm624ELm397ELm31ELj2567483615ELm11ELj4294967295ELm7ELj2636928640ELm15ELj4022730752ELm18ELj1812433253EEER4gridEUlvE_Lb1EE3auxESV_Lb1EE3auxEEEE3runEv
    @     0x7f6e97187457 thread_proxy
    @     0x7f6e8398bea5 start_thread
    @     0x7f6e836b4b0d __clone
    @                0x0 (unknown)
Aborted

Steps to reproduce

gnina -r receptor.pdbqt -l ligand.pdbqt -o out.sdf --log log.txt --cnn_scoring all --config config.txt
I can successfully run the above command to completion with the '--cnn_scoring refinement' option, but when I substitute '--cnn_scoring all' I encounter the issue. The contents of my config file, including a docking site specific to my receptor, are copied below:

center_x = -3
center_y = 12.15
center_z = 21.376
size_x = 12
size_y = 14
size_z = 26
seed = 0 
verbosity = 2

system configuration

Operating system: Ubuntu
CUDA version: 12.2
Python version: 3.9.0
Gnina version: 1.0.2

I'm running on a computing cluster with 1 allocated GPU and 32GB of memory.

[dkoes]

Unfortunately, this is a known problem with CUDNN. It will work if you build without cudnn.

[svolk19-stanford]

Thank you!

[kcgthb]

Hi @dkoes,
When you say "known problem with CUDNN", do that mean an issue in cuDNN, or an issue in gnina when used with cuDNN? If the former, could you please point to the specific cuDNN issue/version you're referring to?
Thanks!

[dkoes]

The later, although as far as I can tell there is nothing wrong with the code, it just stopped working with a cuDNN upgrade. As the plan is to switch to pytorch (next big project after I wrap up covalent docking), fixing it isn't a high priority.