[BUG] DBSCAN.fit: 'pylibraft.common.handle.Handle' object has no attribute 'sync'

[AndrejBazhan]

Describe the bug
exception from fit():
File "/usr/local/lib/python3.10/dist-packages/cuml/internals/api_decorators.py", line 188, in wrapper
ret = func(*args, **kwargs)
File "dbscan.pyx", line 363, in cuml.cluster.dbscan.DBSCAN._fit
AttributeError: 'pylibraft.common.handle.Handle' object has no attribute 'sync'

Steps/Code to reproduce bug
Hope you'll see reason without data exchange.

Expected behavior
no exception

Environment details (please complete the following information):

• Environment location: Bare-metal
• Linux Distro/Architecture: [Ubuntu 22.04 amd64]
• GPU Model/Driver: [A100 and driver 5XX]
• CUDA: [12.2]
• Method of cuDF & cuML install: pip on clean Python 3.10.12

pip install -i https://pypi.org/simple --no-input
--extra-index-url=https://pypi.nvidia.com
cudf-cu12==24.2.* dask-cudf-cu12==24.2.* cuml-cu12==24.2.*
cugraph-cu12==24.2.* cuspatial-cu12==24.2.* cuproj-cu12==24.2.*
cuxfilter-cu12==24.2.* cucim-cu12==24.2.* pylibraft-cu12==24.2.*
raft-dask-cu12==24.2.*

[dantegd]

This is a very weird issue, the sync method clearly is in RAFT's handle https://github.com/rapidsai/raft/blame/335236c705c0c53da8a4bf6a22835fdbe669f1df/python/pylibraft/pylibraft/common/handle.pyx#L113, does this happen consistently with all estimators? The only thing that comes to mind right now is that there might be an old pylibraft somwhere in the system that is taking precedence.

[AndrejBazhan]

This is a very weird issue, the sync method clearly is in RAFT's handle https://github.com/rapidsai/raft/blame/335236c705c0c53da8a4bf6a22835fdbe669f1df/python/pylibraft/pylibraft/common/handle.pyx#L113, does this happen consistently with all estimators? The only thing that comes to mind right now is that there might be an old pylibraft somwhere in the system that is taking precedence.

This is clean install on ubuntu 22.04, python 3.10.12.
Problem is persistent, but depends on data - not all scenarios lead to sync invocation.
Can you check that pip install above doesn't have config problems that lead to installation of an old version of pylibraft?

[AndrejBazhan]

from OS image script:

python3.10

add-apt-repository -y ppa:deadsnakes/ppa
apt-get update
apt-get install -y python3.10 python3.10-distutils libpython3.10-dev
ln -sf /usr/bin/python3.10 /usr/bin/python
ln -sf /usr/bin/python3.10 /usr/bin/python3
python --version

pip

curl -k https://bootstrap.pypa.io/get-pip.py > get_pip.py
python get_pip.py --trusted-host pypi.python.org --trusted-host pypi.python.org
rm get_pip.py
python -m pip --version
python -m pip install --upgrade setuptools
python -m pip install setuptools-rust

cuda

apt-key del 7fa2af80
mkdir cuda
cd cuda

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt-get update
apt-get -y install
cuda-command-line-tools-12-2
cuda-compiler-12-2
cuda-libraries-12-2
cuda-nvrtc-12-2
gds-tools-12-2
libcudnn8=8.9.7.29-1+cuda12.2
libnccl2=2.19.3-1+cuda12.2

export PTXCOMPILER_CHECK_NUMBA_CODEGEN_PATCH_NEEDED="0"
export PTXCOMPILER_KNOWN_DRIVER_VERSION="12.0"
export PTXCOMPILER_KNOWN_RUNTIME_VERSION="12.2"

pip install -i https://pypi.org/simple --no-input cuda-python==12.2.1

rapids

pip install -i https://pypi.org/simple
--extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple
"cudf-cu12>=24.4.0a0,<=24.4" "cuml-cu12>=24.4.0a0,<=24.4"
"cugraph-cu12>=24.4.0a0,<=24.4" "cuspatial-cu12>=24.4.0a0,<=24.4"
"cuproj-cu12>=24.4.0a0,<=24.4" "cucim-cu12>=24.4.0a0,<=24.4"
"pylibraft-cu12>=24.4.0a0,<=24.4" "raft-dask-cu12>=24.4.0a0,<=24.4"
"dask-cuda>=24.4.0a0,<=24.4"