DOC Add example showcasing HGBT regression

[ArturoAmorQ]

Reference Issues/PRs

Fixes #26826. See also #21967 and #23746 on missing values documentation.

What does this implement/fix? Explain your changes.

This PR adds an example to:

• replace the landing-page figure by a simple didactic plot
• showcase HGBT nice features such as:
  • Quantile regression
  • Support of missing values
  • Monotonicity constraints
• be cross-linked in the documentation
• be cross-linked in other examples

Any other comments?

The original issue suggests also demoing support of categorical values, but we already have Categorical Feature Support in Gradient Boosting, which is only linked in the present example as it is a very good example itself.

Indeed, we also have a Monotonic constraints example but it can be merged with the example from this PR.

[github-actions]

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: ea70999. Link to the linter CI: here}

[lorentzenchr]

A first iteration.

[lorentzenchr]

Maybe, we should change this section to highlight early stopping and finding a good pair of (max_iter, learning_rate).
I consider this important as it is shown nowhere and usually the first step to train a HGBT.

[lorentzenchr]

If you need help here, just say so.

[ArturoAmorQ]

Maybe, we should change this section to highlight early stopping and finding a good pair of (max_iter, learning_rate)

I guess you mean by doing a grid search over those 2 parameters and exploring the respective n_iter_ attribute, right? Or what do you have in mind?

[lorentzenchr]

Grid search is overkill, even not so helpful. Just choose one learning rate, fit the HBGT on the training set and determine the best max_iter via early stopping and some explicit validation_fraction. From there on, use this max_iter and don't use early stopping anymore.
Once, this PR and #27124 are merged, we can add cross validation for this step.

[ArturoAmorQ]

I ran into the issue reported in #25460, as early_stopping uses shuffle=True for the internal validation, which is not the right thing to do when dealing with time series, as done in this example.

[lorentzenchr]

Exactly. But for the time being, it is better to "learn" max iter with the current implementation and early stopping than to rely on defaults.
After all, max iter and the learning rate are the most important parameters.

I would just add a comment that it is not optimal for time series ATM.

[ArturoAmorQ]

Addressed in 9a486b8.

[ArturoAmorQ]

I think I have addressed all the comments. Please let me know otherwise.

[glemaitre]

This looks already good to me. I'm just wondering that we don't show the automatic way of dealing with categorical data. The variable day is actually a "category" variable. I'm wondering if we could not simplify modify the code and mention that we have this new way to detecting and encoding the categorical variable internally.

[glemaitre]

I think we need a sentence to describe what we see on the figure. 5 iterations is not enough to be able to predict. But at 50, we are already able to do a good job.

[ArturoAmorQ]

The variable day is actually a "category" variable.

The thing is that day is already ordinal encoded, so I don't think we can demo the categorical variables support here.

[glemaitre]

It does not really matter. Since this is a category columns, the tree will treat it as such.

A potential thing that we could do is to replace the integers by the day as string at the beginning. I don't know if eventually it removes some boilerplate when plotting because we already have the name instead of remapping the integer to the days.

[glemaitre]

This said, I would be happy to merge this PR as-is because it is a net improvement and see if we can further improve it with this increment.

[ArturoAmorQ]

This said, I would be happy to merge this PR as-is because it is a net improvement and see if we can further improve it with this increment.

Should I push my own green button?

[lorentzenchr]

Should I push my own green button?

No. @glemaitre you did the latest review. Fine to merge?

[glemaitre]

Yes. Fine To merge. @lorentzenchr do you have further changes (just because the status is not approved on your side).

[lorentzenchr]

the status is not approved on your side

I approved a month ago and then you requested a review by me.
I‘ll merge.

[glemaitre]

Whoops my bad, I probably misclicked. Sorry. Sent from my iPhoneOn 22 Feb 2024, at 18:53, Christian Lorentzen ***@***.***> wrote: Merged #26991 into main. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[GaelVaroquaux]

Very happy about this! Congratulation to every one involved.

Nitpick: I would remove "advanced" from the title, as I think that many people should look at this example, and not only advanced people.

If y'all agree, I'll send a PR that does nothing else than remove this word 😀 😀

[lorentzenchr]

@GaelVaroquaux Please go on and ping me for quick review if you‘d like.

[GaelVaroquaux]

Here it is #28508

I was not true to my words :$ I removed a tiny bit more than "advanced" :)

Thanks!!