Multiple output regression

[miguelmartin75]

How do I perform multiple output regression? Or is it simply not possible?

My current assumption is that I would have to modify the code-base such that XGMatrix supports a matrix as labels and that I would have to create a custom objective function.

My end goal would be to perform regression to output two variables (a point) and to optimise euclidean loss. Would I be better off to make two seperate models (one for x coordinates and one for y coordinates).

Or... would I be better off using a random forest regressor within sklearn or some other alternative algorithm?

[khotilov]

Multivariate/multilabel regression is not currently implemented #574 #680
Tianqi had added some relevant placeholder data structures to gbtree learner, but no one had time, I guess, to work the machinery out.

[jindongwang]

Pity, since many competitions are with multi-outputs

[MarkusBonsch]

This would be a really nice feature to have.

[joel-thomas-wilson]

Do we have any updates on this?

[hcho3]

I'm adding this feature to the feature request tracker: #3439. Hopefully, we can get to it some point.

[JacobKempster]

I agree - this feature would be extremely valuable (exactly what I need right now...)

[lenselinkbart]

I also agree, while this is quite trivial to do in neural nets, it would be nice to also be able to do this in xgboost.

[cp9612]

Would like to see this feature coming

[veonua]

any reason why it is closed?

[hcho3]

@veonua See #3439.

[loretoparisi]

In the meanwhile there is any alternative, like any ensemble of single output models like:

# Fit a model and predict the lens values from the original features
model = XGBRegressor(n_estimators=2000, max_depth=20, learning_rate=0.01)
model = multioutput.MultiOutputRegressor(model)
model.fit(X_train, X_lens_train)
preds = model.predict(X_test)

from: https://gist.github.com/MLWave/4a3f8b0fee43d45646cf118bda4d202a

[jimmywan]

In the meanwhile there is any alternative, like any ensemble of single output models like:

https://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputRegressor.html

[cmottet]

I am going to also weigh in and say that having such feature would be extremely handy. The MultiOutputRegressor mentioned above is a nice wrapper to build multiple models at once and it does work well for predicting target variables that are independent from one another. However, if the target variables are highly correlated, then you really want to build one model that predicts a vector.

[MxNl]

A year has passed soon since the last comment :-). This is why I want to repeat the wish to have such an interesting feature. I would be happy to see this. Thanks anyway for all your work.

[hcho3]

Reopening for visibility.

[kk26269]

Multivariate/multilabel regression is not currently implemented #574 #680
Tianqi had added some relevant placeholder data structures to gbtree learner, but no one had time, I guess, to work the machinery out.

Hello ,I have used the SckitLearn estimator and passed my script (.py)written for multioutput regression to the same and I could create endpoints.
I have reffered following repo.
https://github.com/qlanners/ml_deploy/tree/master/aws/scikit-learn/sklearn_estimators_locally.
Changes done are :

Y= dataset.iloc[:,-3:]
X = dataset.iloc[:,:-3]
X_train, X_test , Y_train , Y_test =train_test_split(X, Y , test_size = 0.20,random_state =100)

gbr = GradientBoostingRegressor()
modelMOR = MultiOutputRegressor(estimator=gbr)
modelMOR.fit(X_train, Y_train)

[mirik123]

The MultiOutputRegressor is bad alternative because it doesn't update eval_set dataset together with the main train (X, y) dataset.

[trivialfis]

I would love to spend some time on this ...

[loretoparisi]

I would love to spend some time on this ...

I have used this approach and it seems to work fine

#2087 (comment)

[StatMixedML]

Is there any update on this? Can we make it a joint effort to have the multioutput regression available. Irrespective of the independence modelling of several responses/y-variables, it would be great to have the xgb.DMatrix accept a list or a np.array with shape >1.

[jameslamb]

To be honest, I also am not sure whether it's exactly the same. But it should be similar, right? Whether you are working on multiple tasks like "regression and classification" or multiple targets like "regression predicting y_1 and y_2", you still are in a situation like "find splits that balance gain across multiple loss functions".

To be honest, I haven't read the paper and am not planning to actively work on this (we have many other higher priorities in LightGBM right now).

[StatMixedML]

@jameslamb

To be honest, I haven't read the paper and am not planning to actively work on this (we have many other higher priorities in LightGBM right now).

Sure, I understand that. I am not sure I find the time either. So maybe let's pause this and see if the community picks it up.

[trivialfis]

Did a quick scan over a couple of papers. I don't have a good understanding of various algorithms yet, but vector leaf seems to be the essential component of all proposed methods. I will try to prioritize it and share a roadmap for a path forward.

[StatMixedML]

@trivialfis Ok nice! Looking forward

[StatMixedML]

@trivialfis This might be an interesting approach to incorporate into XGBoost

SketchBoost: Fast Gradient Boosted Decision Tree for Multioutput Problems

The paper says

Moreover, the proposed methods are easy to implement upon modern boosting frameworks such as XGBoost

You find the code here https://github.com/sb-ai-lab/Py-Boost

[trivialfis]

@StatMixedML Thank you for the references. Here's an early version of vector-leaf: #8616 No specific optimization yet.

[StatMixedML]

@trivialfis Very nice, I`ll have a look into it

[trivialfis]

I will cleanup the code in the coming days. There are some known issues that break the existing code, the only thing that's working is the demo at the moment. It's for discussion and far from ready.

[StatMixedML]

@trivialfis Sure, take your time. Let me know once I can use it.

Looking forward to it!

[StatMixedML]

@trivialfis I have seen that you created a PR for a first version of the multi-target tree. This is awesome!!

Let me know once I can test it. Would be great to run some examples and compare accuracy and runtime. Willing to volunteer on this!

[lcrmorin]

I am currently trying this. Should I expect any performance / memory gain over tuning multiple models ?

[trivialfis]

Hi @StatMixedML @lcrmorin, thank you for volunteering! The PR is not ready yet, I still need to figure out some parts of the parameter interface and do more tests. If you really want to try to code, the demo/guide-python/multioutput_regression.py would be a good starting place, See the rmse_model function and parameters used in there.

[StatMixedML]

@lcrmorin So the advantage of using multi-output models is that you don't have to train a separate model for each response variable. Also, as outlined in Multi-Target XGBoostLSS Regression, you can model dependencies between the different responses. What @trivialfis is currently working on is to speed-up the estimation using multi-target trees to better and efficiently scale to multiple response variables.

[loretoparisi]

Wow! Shared
https://twitter.com/loretoparisi/status/1623958275279519745?s=46&t=bN2GaGENwqbkg0nK-MY7tg

[lcrmorin]

So that would also help inference time, right ?

[StatMixedML]

@lcrmorin We would expect to see the highest efficiency gains during training time, especially for HPO / cross-validation.

[trivialfis]

The bare bone implementation is merged, please help test out. :-)
You can find reference for nightly build in xgboost's Python installation document. The computation performance is not yet optimized, please expect some quirks.

[trivialfis]

The link will be available once the CI passes

[trivialfis]

A bug fix PR for prediction along with a small optimization: #8968 .

[lcrmorin]

Just being curious; would this allows some missing / masked targets ? (I have multi-targets time-series applications in mind, where longer horizons targets are not available immediatly).

[trivialfis]

Not planned at the moment, the label is required to be dense. But I will mark that as a feature request and see if we can find a way to train boosting tree models with missing labels.

[trivialfis]

Hi all, thank you for joining the discussion and the helpful feedbacks! Let's continue the discussion in #9043 .