Save training and vaildation loss in loss_curve_ in MLPClassifier and MLPRegressor with early_stopping

[pplonski]

Let's select MLPClassifier.

In MLPClassifier there is loss_curve_ available. If there is early_stopping enabled then some part of the data is used as validation. Can we save the loss of training and validation data in the loss_curve_ as well?

Additional context

I've compared the MLP implementation with Tensorflow implementation and it works very well, there are no significant differences in the performance. You can read the comparison details at my blog post. I'm using the MLP in my AutoML mljar-supervised which creates Markdown reports for each model. I would like to have learning curves available in the report.

[glemaitre]

This is related to a more general API question on how to include callback: #16925
Basically, callbacks would allow storing losses and should be generic enough for the different learners.

[pplonski]

Having callbacks will be super helpful. Do you know when they will be available?

In the current implementation, the loss curve is already available, but with loss values from train samples. In the case of early stopping, there should be a quick way to add validation loss as well.

[glemaitre]

Having callbacks will be super helpful. Do you know when they will be available?

I personally did not follow the advancement of this feature and if there is any blocker regarding the API.
It will be probably too soon to expect it in 0.24 since we should release pretty soon but it might be a milestone for 0.25 then.

[NicolasHug]

As a side note, we've decided to stop supporting additional features for the MLP module. I'd recommend to just use PyTorch or TF instead

[pplonski]

That's a strange decision, sklearn MLP works pretty well. I did a comparison of MLP from sklearn vs Keras+TF. Sklearn MLP performs very well and was faster on CPU computations. Check the comparison here: https://mljar.com/blog/tensorflow-vs-scikit-learn/ Not all NN must be deep on computed on GPU.