Unable to install BERTopic due to fail in building HDBSCAN on Kaggle Notebooks.

[dafajon]

• Also having similar issues in python-3.7-buster and python-3.9-buster images in other projects.

Environment: Kaggle

Error Message:

error: subprocess-exited-with-error
  
  × Building wheel for hdbscan (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [168 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/hdbscan
      copying hdbscan/validity.py -> build/lib.linux-x86_64-cpython-310/hdbscan
      copying hdbscan/__init__.py -> build/lib.linux-x86_64-cpython-310/hdbscan
      copying hdbscan/hdbscan_.py -> build/lib.linux-x86_64-cpython-310/hdbscan
      copying hdbscan/robust_single_linkage_.py -> build/lib.linux-x86_64-cpython-310/hdbscan
      copying hdbscan/prediction.py -> build/lib.linux-x86_64-cpython-310/hdbscan
      copying hdbscan/plots.py -> build/lib.linux-x86_64-cpython-310/hdbscan
      copying hdbscan/flat.py -> build/lib.linux-x86_64-cpython-310/hdbscan
      creating build/lib.linux-x86_64-cpython-310/hdbscan/tests
      copying hdbscan/tests/test_prediction_utils.py -> build/lib.linux-x86_64-cpython-310/hdbscan/tests
      copying hdbscan/tests/__init__.py -> build/lib.linux-x86_64-cpython-310/hdbscan/tests
      copying hdbscan/tests/test_hdbscan.py -> build/lib.linux-x86_64-cpython-310/hdbscan/tests
      copying hdbscan/tests/test_flat.py -> build/lib.linux-x86_64-cpython-310/hdbscan/tests
      copying hdbscan/tests/test_rsl.py -> build/lib.linux-x86_64-cpython-310/hdbscan/tests
      running build_ext
      /tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-rpv8fbne/hdbscan_3fb6ecdb1029489c8158cac23313e5ba/hdbscan/_hdbscan_tree.pyx
        tree = Parsing.p_module(s, pxd, full_module_name)
      Compiling hdbscan/_hdbscan_tree.pyx because it changed.
      [1/1] Cythonizing hdbscan/_hdbscan_tree.pyx
      building 'hdbscan._hdbscan_tree' extension
      creating build/temp.linux-x86_64-cpython-310
      creating build/temp.linux-x86_64-cpython-310/hdbscan
      gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/conda/include -fPIC -O2 -isystem /opt/conda/include -fPIC -I/opt/conda/include/python3.10 -I/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/numpy/core/include -c hdbscan/_hdbscan_tree.c -o build/temp.linux-x86_64-cpython-310/hdbscan/_hdbscan_tree.o
      In file included from /tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                       from /tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                       from /tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                       from hdbscan/_hdbscan_tree.c:1097:
      /tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
         17 | #warning "Using deprecated NumPy API, disable it with " \
            |  ^~~~~~~
      gcc -pthread -B /opt/conda/compiler_compat -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib build/temp.linux-x86_64-cpython-310/hdbscan/_hdbscan_tree.o -o build/lib.linux-x86_64-cpython-310/hdbscan/_hdbscan_tree.cpython-310-x86_64-linux-gnu.so
      /tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-rpv8fbne/hdbscan_3fb6ecdb1029489c8158cac23313e5ba/hdbscan/_hdbscan_linkage.pyx
        tree = Parsing.p_module(s, pxd, full_module_name)
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
      import numpy as np
      cimport numpy as np
      
      from libc.float cimport DBL_MAX
      
      from dist_metrics cimport DistanceMetric
      ^
      ------------------------------------------------------------
      
      hdbscan/_hdbscan_linkage.pyx:12:0: 'dist_metrics.pxd' not found
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
      import numpy as np
      cimport numpy as np
      
      from libc.float cimport DBL_MAX
      
      from dist_metrics cimport DistanceMetric
      ^
      ------------------------------------------------------------
      
      hdbscan/_hdbscan_linkage.pyx:12:0: 'dist_metrics/DistanceMetric.pxd' not found
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
      
      
      cpdef np.ndarray[np.double_t, ndim=2] mst_linkage_core_vector(
              np.ndarray[np.double_t, ndim=2, mode='c'] raw_data,
              np.ndarray[np.double_t, ndim=1, mode='c'] core_distances,
              DistanceMetric dist_metric,
              ^
      ------------------------------------------------------------
      
      hdbscan/_hdbscan_linkage.pyx:58:8: 'DistanceMetric' is not a type identifier
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
                      continue
      
                  right_value = current_distances[j]
                  right_source = current_sources[j]
      
                  left_value = dist_metric.dist(&raw_data_ptr[num_features *
                                                ^
      ------------------------------------------------------------
      
      hdbscan/_hdbscan_linkage.pyx:129:42: Cannot convert 'double_t *' to Python object
      
      Error compiling Cython file:
      ------------------------------------------------------------
      ...
                  right_value = current_distances[j]
                  right_source = current_sources[j]
      
                  left_value = dist_metric.dist(&raw_data_ptr[num_features *
                                                              current_node],
                                                &raw_data_ptr[num_features * j],
                                                ^
      ------------------------------------------------------------
      
      hdbscan/_hdbscan_linkage.pyx:131:42: Cannot convert 'double_t *' to Python object
      Compiling hdbscan/_hdbscan_linkage.pyx because it changed.
      [1/1] Cythonizing hdbscan/_hdbscan_linkage.pyx
      Traceback (most recent call last):
        File "/opt/conda/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/opt/conda/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/opt/conda/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 416, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 401, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 487, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 96, in <module>
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 343, in run
          self.run_command("build")
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "<string>", line 26, in run
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
          self._build_extensions_serial()
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
          self.build_extension(ext)
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/Cython/Distutils/build_ext.py", line 122, in build_extension
          new_ext = cythonize(
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
          cythonize_one(*args)
        File "/tmp/pip-build-env-mjuo72sw/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
          raise CompileError(None, pyx_file)
      Cython.Compiler.Errors.CompileError: hdbscan/_hdbscan_linkage.pyx
      [end of output]

[manjunath-hanmantgad]

Same error with py 3.10.11