GroupKFold

[lukegriffiths]

Hi - is it possible to use a Group KFold cross-validation scheme in MapieClassifier? Or even a custom cross-validation scheme? For my use case, I need to split up my data by group because data points within a group can be very similar. All examples and documentation I have seen so far point to using KFold or LeaveOneOut which doesn't work in this case.

[thibaultcordier]

Hello @lukegriffiths,

This is indeed a suggestion that was put forward in issue #202 but has not yet been addressed. I agree with you that a customised cross-validation scheme would be welcome. In practice, this requires that parameters to the split method (such as group=...) can be passed from the fit method.

When you say "I need to split up my data by group because data points within a group can be very similar", are you referring to a paper that uses this splitting technique? Because if you group similar points together, aren't you afraid of having models that overfit?

I'm curious to read your answer.

[lukegriffiths]

Thanks Thibault for the reply. Sorry I didn't see the other issue which would indeed address my use case. There is geospatial structure in my dataset so if I randomly sample across all data and locations then I can end up with very similar data points from a same given location in both my train and test set. GroupKfold for CV would help avoid this. Hope that makes sense.