[BUG] Random Forest with Pycaret on gpu throws exception : Could not load the correct number of nodes

[francoisMunger]

Describe the bug
Random Forest with Pycaret on gpu throws exception : Could not load the correct number of nodes

Steps/Code to reproduce bug
Install Pycaret rapidsai and cuml using conda

Code

import pandas as pd
from imblearn.over_sampling import ADASYN
from pycaret.classification import automl, compare_models, finalize_model, pull, setup, create_model
import pycaret_extensions as pycaret_ext

def main():
    df = pd.read_csv("data/hour.csv", sep=",").drop_duplicates(subset=None).dropna()
    df["cnt"] = pd.cut(df["cnt"], bins=[0, 500, 999], labels=[0, 1]).astype("int")
    setup(
        data=df,
        target=-1,  
        normalize=True,
        normalize_method="robust",
        transformation=True,
        transformation_method="quantile",
        fix_imbalance=False,
        fix_imbalance_method=ADASYN(sampling_strategy="minority"),  
        remove_multicollinearity=True,  
        train_size=0.75,  
        fold=3, 
        log_experiment=False,  
        session_id=1,  
        system_log=True,
        use_gpu=True,
    )
    names = ["rf"]
	
    top5: list[object] = compare_models(n_select=5, sort="Accuracy",include=names)

Result

Processing: 0%| | 0/9 [00:00<?, ?it/s]
Then the program stops. I was able to get the exception text while debugging only.

The dataset used (hour.csv) is freely available at https://rubikscode.net/2021/07/19/top-23-best-public-datasets-for-practicing-machine-learning/
under Bike Sharing Demand Dataset. I tried with Wine quality dataset too and got same bug.

Expected behavior
Processing continue as expected.
Note : The code works if you remove the use_gpu flag in setup().

Environment details (please complete the following information):

See file for env details
env.txt

Additional context
Stack Trace :
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/metrics/_scorer.py", line 76, in _cached_call
return cache[method]
KeyError: 'predict'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/pycaret_experiment/supervised_experiment.py", line 794, in compare_models
model, model_fit_time = self._create_model(**create_model_args)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/pycaret_experiment/supervised_experiment.py", line 1533, in _create_model
model, model_fit_time, model_results, _ = self._create_model_with_cv(
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/pycaret_experiment/supervised_experiment.py", line 1126, in _create_model_with_cv
scores = cross_validate(
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/mlflow/utils/autologging_utils/safety.py", line 578, in safe_patch_function
patch_function(call_original, *args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/mlflow/sklearn/init.py", line 1875, in patched_fn_with_autolog_disabled
return original(*args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/mlflow/utils/autologging_utils/safety.py", line 559, in call_original
return call_original_fn_with_event_logging(_original_fn, og_args, og_kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/mlflow/utils/autologging_utils/safety.py", line 494, in call_original_fn_with_event_logging
original_fn_result = original_fn(og_args, **og_kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/mlflow/utils/autologging_utils/safety.py", line 556, in _original_fn
original_result = original(_og_args, **_og_kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 266, in cross_validate
results = parallel(
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/utils/parallel.py", line 63, in call
return super().call(iterable_with_config)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/parallel.py", line 1085, in call
if self.dispatch_one_batch(iterator):
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/parallel.py", line 901, in dispatch_one_batch
self._dispatch(tasks)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/parallel.py", line 819, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 208, in apply_async
result = ImmediateResult(func)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/_parallel_backends.py", line 597, in init
self.results = batch()
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/parallel.py", line 288, in call
return [func(*args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/joblib/parallel.py", line 288, in
return [func(*args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/utils/parallel.py", line 123, in call
return self.function(*args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/patches/sklearn.py", line 124, in fit_and_score
return wrapper(*args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/patches/sklearn.py", line 122, in wrapper
return _fit_and_score(*args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 708, in _fit_and_score
test_scores = _score(estimator, X_test, y_test, scorer, error_score)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/patches/sklearn.py", line 144, in wrapper
return f(args[0], *tuple(args[1:]), **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/model_selection/_validation.py", line 767, in _score
scores = scorer(estimator, X_test, y_test)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/metrics/_scorer.py", line 121, in call
raise e
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/metrics/_scorer.py", line 115, in call
score = scorer._score(cached_call, estimator, *args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/metrics/_scorer.py", line 276, in _score
y_pred = method_caller(estimator, "predict", X)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/sklearn/metrics/_scorer.py", line 78, in _cached_call
result = getattr(estimator, method)(*args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/pipeline.py", line 327, in predict
y = self.steps[-1][-1].predict(X, **predict_params)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 190, in wrapper
return func(*args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/cuml_wrappers.py", line 130, in predict
X = super().predict(X, *args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 188, in wrapper
ret = func(*args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/nvtx/nvtx.py", line 115, in inner
result = func(*args, **kwargs)
File "randomforestclassifier.pyx", line 605, in cuml.ensemble.randomforestclassifier.RandomForestClassifier.predict
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/cuml/internals/api_decorators.py", line 188, in wrapper
ret = func(*args, **kwargs)
File "randomforest_common.pyx", line 348, in cuml.ensemble.randomforest_common.BaseRandomForestModel._predict_model_on_gpu
File "randomforest_common.pyx", line 216, in cuml.ensemble.randomforest_common.BaseRandomForestModel._obtain_treelite_handle
File "randomforest_shared.pyx", line 120, in cuml.ensemble.randomforest_shared.treelite_deserialize
File "randomforest_shared.pyx", line 104, in cuml.ensemble.randomforest_shared.init_from_frames
File "randomforest_shared.pyx", line 83, in cuml.ensemble.randomforest_shared._init_from_frames
RuntimeError: Could not load the correct number of nodes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/datadrive/src/train_automl.py", line 92, in
main()
File "/datadrive/src/train_automl.py", line 68, in main
top5: list[object] = compare_models(n_select=5, sort="Accuracy", errors="raise")
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/utils/generic.py", line 965, in wrapper
return func(*args, **kwargs)
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/classification/functional.py", line 814, in compare_models
return _CURRENT_EXPERIMENT.compare_models(
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/classification/oop.py", line 1180, in compare_models
return_values = super().compare_models(
File "/datadrive/miniconda3/envs/ia-bootstrap-gpu/lib/python3.10/site-packages/pycaret/internal/pycaret_experiment/supervised_experiment.py", line 806, in compare_models
raise RuntimeError(
RuntimeError: create_model() failed for model rf. RuntimeError: Could not load the correct number of nodes

[dantegd]

Thanks for the issue @francoisMunger, I'm not sure why this is failing, it could be a bug with the integration of pycaret and cuml with recent versions, will try to reproduce and solve the issue