An error occurred when using the xgboost as a classifier for hiclass

[RamSnoussi]

Hi,
Bellow it's my example when using the xgboost classifier for hiclass. My question is specifically directed to the hiClass Python package for hierarchical classification. I would like to model the problem using hierarchical classification approach to proceed like in figure below:

How can I correct this error?

from hiclass import LocalClassifierPerParentNode
from xgboost import XGBClassifier
import numpy as np

X_train = np.array([
    [40.7,  1. ,  1. ,  2. ,  5. ,  2. ,  1. ,  5. , 34.3],
    [39.2,  0. ,  2. ,  4. ,  1. ,  3. ,  1. ,  2. , 34.1],
    [40.6,  0. ,  3. ,  1. ,  4. ,  5. ,  0. ,  6. , 27.7],
    [36.5,  0. ,  3. ,  1. ,  2. ,  2. ,  0. ,  2. , 39.9],
])

Y_train = np.array([
    ['Gastrointestinal', 'Norovirus', ''],
    ['Respiratory', 'Covid', ''],
    ['Allergy', 'External', 'Bee Allergy'],
    ['Respiratory', 'Cold', ''],
])
X_test = np.array([[35.5,  0. ,  1. ,  1. ,  3. ,  3. ,  0. ,  2. , 37.5]])
classifier = LocalClassifierPerParentNode(local_classifier=XGBClassifier(objective='multi:softmax'))
classifier.fit(X_train, Y_train)
predictions = classifier.predict(X_test)
print(predictions)

The output of the program:
2024-05-07 14:19:52,005 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Gastrointestinal'
2024-05-07 14:19:52,005 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Gastrointestinal'
2024-05-07 14:19:52,005 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Gastrointestinal'
2024-05-07 14:19:52,005 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Gastrointestinal'
2024-05-07 14:19:52,005 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Gastrointestinal'
2024-05-07 14:19:52,005 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Gastrointestinal'
2024-05-07 14:19:52,005 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Gastrointestinal'
2024-05-07 14:19:52,005 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Gastrointestinal'
2024-05-07 14:19:52,005 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Gastrointestinal'
2024-05-07 14:19:52,005 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Gastrointestinal'
2024-05-07 14:19:52,024 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Norovirus'
2024-05-07 14:19:52,024 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Norovirus'
2024-05-07 14:19:52,024 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Norovirus'
2024-05-07 14:19:52,024 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Norovirus'
2024-05-07 14:19:52,024 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Norovirus'
2024-05-07 14:19:52,024 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Norovirus'
2024-05-07 14:19:52,024 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Norovirus'
2024-05-07 14:19:52,024 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Norovirus'
2024-05-07 14:19:52,024 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Norovirus'
2024-05-07 14:19:52,024 - LCPPN - WARNING - Fitting ConstantClassifier for node 'Norovirus'
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[15], line 24
     21 classifier = LocalClassifierPerParentNode(local_classifier=XGBClassifier(objective='multi:softmax'))
     23 # Train local classifier per node
---> 24 classifier.fit(X_train, Y_train)
     26 # Predict
     27 predictions = classifier.predict(X_test)

File ~/anaconda3/lib/python3.9/site-packages/hiclass/LocalClassifierPerParentNode.py:109, in LocalClassifierPerParentNode.fit(self, X, y)
    106 self._initialize_local_classifiers()
    108 # Fit local classifiers in DAG
--> 109 self._fit_digraph()
    111 # TODO: Store the classes seen during fit
    112 
    113 # TODO: Add function to allow user to change local classifier
   (...)
    120 
    121 # Delete unnecessary variables
    122 self._clean_up()

File ~/anaconda3/lib/python3.9/site-packages/hiclass/LocalClassifierPerParentNode.py:348, in LocalClassifierPerParentNode._fit_digraph(self)
    346     self.hierarchy_.nodes[node]["classifier"] = ConstantClassifier()
    347     classifier = self.hierarchy_.nodes[node]["classifier"]
--> 348 classifier.fit(X, y)

File ~/anaconda3/lib/python3.9/site-packages/xgboost/core.py:730, in require_keyword_args.<locals>.throw_if.<locals>.inner_f(*args, **kwargs)
    728 for k, arg in zip(sig.parameters, args):
    729     kwargs[k] = arg
--> 730 return func(**kwargs)

File ~/anaconda3/lib/python3.9/site-packages/xgboost/sklearn.py:1471, in XGBClassifier.fit(self, X, y, sample_weight, base_margin, eval_set, eval_metric, early_stopping_rounds, verbose, xgb_model, sample_weight_eval_set, base_margin_eval_set, feature_weights, callbacks)
   1466     expected_classes = self.classes_
   1467 if (
   1468     classes.shape != expected_classes.shape
   1469     or not (classes == expected_classes).all()
   1470 ):
-> 1471     raise ValueError(
   1472         f"Invalid classes inferred from unique values of `y`.  "
   1473         f"Expected: {expected_classes}, got {classes}"
   1474     )
   1476 params = self.get_xgb_params()
   1478 if callable(self.objective):

ValueError: Invalid classes inferred from unique values of `y`.  Expected: [0 1], got ['Respiratory::HiClass::Separator::Cold'
 'Respiratory::HiClass::Separator::Covid']

[tcsmaster]

If you want to use the softmax objective, you have to encode your label to the range [0, num_classes), which you can't do inside hiclass.

[mirand863]

Hi @RamSnoussi,

Thank you for the interest in HiClass. As mentioned by @tcsmaster, you need to encode your labels. Here is an example of how to do it, but you have to be careful to call the method fit to encode all of them for all the levels at the same time and then you can encode each level individually with the method encode. I remember doing this encoding previously, so if you want I can look up and see if I can find my old code to share with you.

[tcsmaster]

If not @RamSnoussi, then I would really appreciate the code snippet.

[mirand863]

If not @RamSnoussi, then I would really appreciate the code snippet.

The snippet I have is not well structured, but the algorithm goes like this:

from sklearn.preprocessing import LabelEncoder


np_y = np.array(y)  # convert y to a numpy array if it is not yet
flat_y = np.unique(np.append(np_y.flatten(), "hiclass::root"))  # flatten and return all unique labels from the hierarchy

# encode labels in the hierarchy
label_encoder = LabelEncoder()
label_encoder.fit(flat_y)
y = np.array(
    [label_encoder.transform(row) for row in np_y]
)

Then you can train the hierarchical classifier with the encoded labels and decode the labels after prediction with the method inverse_transform. Hope this helps :)

The code is available in this branch if you want to take a further look

[tcsmaster]

Thank you for the code. Does this also mean that the model needs to encounter all available labels during training?

[RamSnoussi]

Hi @mirand863,
Despite the coding of the labels at the beginning, I think that adding separator '::HiClass::Separator::'causes the problem. Line 214 in 'LocalClassifierPerParentNode.py'

[mirand863]

Thank you for the code. Does this also mean that the model needs to encounter all available labels during training?

Hi @tcsmaster,

Yes, the model needs to see as many labels as possible during training. Just be careful to not leak data in case you have to split between training/test data. We can also discuss this in private. Please, feel free to email me at Fabio.MalcherMiranda@hpi.de

[mirand863]

Hi @mirand863, Despite the coding of the labels at the beginning, I think that adding separator '::HiClass::Separator::'causes the problem. Line 214 in 'LocalClassifierPerParentNode.py'

Hi @RamSnoussi,

Can you please clarify what is the issue with the separator? I was able to execute this code without errors.

[RamSnoussi]

Hi @mirand863,
The problem here is how encoding the separtor ' ::HiClass::Separator:: ' added implicitly by the hiclass? for example, the LCPPN in the 2nd level has ' Respiratory::HiClass::Separator::Covid ' as the target label. If I use the LabelEncoder for each label in Y_Train like this ' Respiratory ' as 0 and ' Covid ' as 1. During training, this label becomes ' 0::::HiClass::Separator::1 ' and it causes an error.