Intermittent RDKit errors when importing latest deepchem version on Windows 11

[z-q-y]

🐛 Bug

Now, every second time I try to import RDkit on VS Code on a Win 11 PC (with CUDA 12.1 enabled), the following error message appears. The error is not always caused by loading RDkit; sometimes it's caused by a bz2 or pandas-related dll. When I restart my kernel the problem goes away on its own 50% of the time.

Input: import deepchem as dc
Output:

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[1], line 1
----> 1 import deepchem as dc
      2 import numpy as np
      3 import pandas as pd

File c:\Users\Ernest.Zhang\AppData\Local\miniforge3\envs\dc272_py39_pt_win\lib\site-packages\deepchem\__init__.py:8
      5 # If you push the tag, please remove `.dev`
      6 __version__ = '2.7.2.dev'
----> 8 import deepchem.data
      9 import deepchem.feat
     10 import deepchem.hyper

File c:\Users\Ernest.Zhang\AppData\Local\miniforge3\envs\dc272_py39_pt_win\lib\site-packages\deepchem\data\__init__.py:16
     14 from deepchem.data.datasets import densify_features
     15 from deepchem.data.supports import *
---> 16 from deepchem.data.data_loader import DataLoader
     17 from deepchem.data.data_loader import CSVLoader
     18 from deepchem.data.data_loader import UserCSVLoader

File c:\Users\Ernest.Zhang\AppData\Local\miniforge3\envs\dc272_py39_pt_win\lib\site-packages\deepchem\data\data_loader.py:16
     14 from deepchem.utils.typing import OneOrMany
     15 from deepchem.utils.data_utils import load_image_files, load_csv_files, load_json_files, load_sdf_files, unzip_file
---> 16 from deepchem.feat import UserDefinedFeaturizer, Featurizer
     17 from deepchem.data import Dataset, DiskDataset, NumpyDataset, ImageDataset
     18 from deepchem.feat.molecule_featurizers import OneHotFeaturizer

File c:\Users\Ernest.Zhang\AppData\Local\miniforge3\envs\dc272_py39_pt_win\lib\site-packages\deepchem\feat\__init__.py:21
     18 from deepchem.feat.binding_pocket_features import BindingPocketFeaturizer
     20 # molecule featurizers
---> 21 from deepchem.feat.molecule_featurizers import AtomicCoordinates
     22 from deepchem.feat.molecule_featurizers import BPSymmetryFunctionInput
     23 from deepchem.feat.molecule_featurizers import CircularFingerprint

File c:\Users\Ernest.Zhang\AppData\Local\miniforge3\envs\dc272_py39_pt_win\lib\site-packages\deepchem\feat\molecule_featurizers\__init__.py:10
      8 from deepchem.feat.molecule_featurizers.maccs_keys_fingerprint import MACCSKeysFingerprint
      9 from deepchem.feat.molecule_featurizers.mordred_descriptors import MordredDescriptors
---> 10 from deepchem.feat.molecule_featurizers.mol2vec_fingerprint import Mol2VecFingerprint
     11 from deepchem.feat.molecule_featurizers.one_hot_featurizer import OneHotFeaturizer
     12 from deepchem.feat.molecule_featurizers.sparse_matrix_one_hot_featurizer import SparseMatrixOneHotFeaturizer

File c:\Users\Ernest.Zhang\AppData\Local\miniforge3\envs\dc272_py39_pt_win\lib\site-packages\deepchem\feat\molecule_featurizers\mol2vec_fingerprint.py:5
      2 from typing import Optional
      4 import numpy as np
----> 5 from rdkit.Chem import AllChem
      6 from deepchem.utils import download_url, get_data_dir, untargz_file
      7 from deepchem.utils.typing import RDKitMol

File c:\Users\Ernest.Zhang\AppData\Local\miniforge3\envs\dc272_py39_pt_win\lib\site-packages\rdkit\__init__.py:24
     20                 if os.path.isfile(lib_path) and not kernel32.LoadLibraryExW(ctypes.c_wchar_p(lib_path), None, 0x00000008):
     21                     raise OSError('Error loading {}; {}'.format(lib, ctypes.FormatError(ctypes.get_last_error())))
---> 24 _delvewheel_patch_1_5_2()
     25 del _delvewheel_patch_1_5_2
     26 # end delvewheel patch

File c:\Users\Ernest.Zhang\AppData\Local\miniforge3\envs\dc272_py39_pt_win\lib\site-packages\rdkit\__init__.py:21, in _delvewheel_patch_1_5_2()
     19 kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
     20 if os.path.isfile(lib_path) and not kernel32.LoadLibraryExW(ctypes.c_wchar_p(lib_path), None, 0x00000008):
---> 21     raise OSError('Error loading {}; {}'.format(lib, ctypes.FormatError(ctypes.get_last_error())))

OSError: Error loading RDKitGenericGroups-8d0c10daf596521a48b94d413753b5be.dll; The operation completed successfully.

To Reproduce

Steps to reproduce the behavior:

1. I created my conda environment by first installing py3.9_cuda12.1_cudnn8_0 and cuda tools from the pytorch and nvidia channels respectively. Original command:

conda create -n dc272_py39_pt_win -c pytorch -c nvidia -c dglteam python=3.9 pytorch pytorch-cuda=12.1 ^
cudatoolkit dgl dgllife "scipy<=1.10" scikit-learn "scipy<=1.10" numpy pandas

2. RDkit was then installed with pip as a dependency of the deepchem package, because I needed a nightly version of deepchem that was only available on pip.

Environment

• OS: Windows 11　22H2　22621.3007
• Python version: 3.9.15 (Channel: conda-forge)
• DeepChem version: 2.7.2.dev (Channel: pip)
• RDKit version: 2023.9.4 (Channel: pip)
• TensorFlow version: 2.10.0 mkl_py39ha510bab_0
• PyTorch version: 2.1.2 py3.9_cuda12.1_cudnn8_0 (Channel: pytorch)
• cudatoolkit version: 11.5.0 (Channel: nvidia)

[z-q-y]

According to the answer to the closest-related problem I can find, this error seems to be Windows' way of saying there are too many (over 10000) handles associated with the python.exe process. In other words, a resource leak happened. However, I'm seeing this error when as little as 1/2 of my 2.5 GHz CPU capacity and 9 GB of my 16 GBs of RAM are occupied.

With this in mind, the question becomes: does importing deepchem cause a resource usage spike?

[arunppsg]

I don't think so. Could you try importing rdkit alone as from rdkit.Chem import AllChem? Because the error seems to arise form rdkit.