Q: call RandomForestClassifier.fit(X, y) multiple times on different (X, y), will it continue training, or will it start over?

[mw66]

Describe the issue linked to the documentation

https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier.fit

Hi, suppose:

r = RandomForestClassifier(...)

r.fit(X1, y1)
r.fit(X2, y2)  # will this continue the model training, or will it start over from scratch again?

What I want to achieve is:

r.fit(X1 + X2, y1 + y2)

But in my app, the (Xn, yn) has to be generated in different loop.

Can we add this clarification to the doc?

Thanks

[alfaro96]

Hi @mingwugmail, thanks for reaching out. You are asking a usage question. The issue tracker is for bugs and new features. GitHub discussions are now enabled for the scikit-learn repository. I am going to close this issue then and converting it to a discussion: feel free to follow-up there.

[NicolasHug]

r.fit(X1, y1)
r.fit(X2, y2)  # will this continue the model training, or will it start over from scratch again?

This will start over from scratch

What I want to achieve is: r.fit(X1 + X2, y1 + y2)

The correct way to do this would be to call r.partial_fit(X1, y1) and then partial_fit(X2, y2). Unfortunately, RandomForest and tree estimators in general don't support partial_fit.

You may be tempted to try this:

rf = RandomForest(n_estimators=100, warm_start=True)
rf.fit(X1, y1)
rf.set_params(n_estimators=150)
rf.fit(X2, y2)

which will fit 100 trees on X1, y1 and then 50 extra trees on X2, y2. In general though, it's incorrect to warm-start on some data and then continue on some other data:warm-start is for (same data, different parameters) while partial_fit is for (same parameters - different data). I wonder whether this is something that might be acceptable with RFs however, due to the bootstrapping of the samples.

[mw66]

I wonder whether this is something that might be acceptable with RFs however, due to the bootstrapping of the samples.

Can we add this partial_fit feature when bootstrap=False?

Also I saw this discussion:

https://stats.stackexchange.com/a/354339

Can you endorse this selected answer?

BTW, can we also add a new attribute:

n_unique_samples_: int

    The number of unique samples (seen so far) when fit is performed (esp. when fit is called multiple times).

[NicolasHug]

Can we add this partial_fit feature when bootstrap=False?

Whether trees can support partial_fit doesn't depend on boostraping. The learning algorithm that is used to train the trees in scikit-learn just isn't designed to support incremental learning, it needs the entire data from the start. There are some discussion in #18888 and follow-up PR

Can you endorse this selected answer?

From a quick look, yes

BTW, can we also add a new attribute:

I'm not sure I understand what this parameter does?

[mw66]

I'm not sure I understand what this parameter does?

I have incremental training in mind, so n_unique_samples_ reports how many samples the trained model have seen so far.

[NicolasHug]

oh right you said attribute not parameter, my bad