Custom version of MinMaxScaler

[Genius1237]

I am looking to modify the functionality of sklearn.preprocessing.MinMaxScaler. In the vanilla implementation, min and max values are calculated individually for each feature and used to perform the scaling. I want to modify this such that the user can specify a set of features such that a common min and max values are calculated for these features.

I have created a custom scaler and have put it's code below. It seems to work for me, but I am not sure if I've done everything correctly. In my code common_groups is a list of lists, where each list in it specifies the which features (by their indices) to calculate the min/max values together for.

class CustomMinMaxScaler(sklearn.preprocessing.MinMaxScaler):
    def _handle_zeros_in_scale(scale, copy=True):
        # if we are fitting on 1D arrays, scale might be a scalar
        if np.isscalar(scale):
            if scale == .0:
                scale = 1.
            return scale
        elif isinstance(scale, np.ndarray):
            if copy:
                # New array to avoid side-effects
                scale = scale.copy()
            scale[scale == 0.0] = 1.0
            return scale

    def __init__(self, common_feats=None, **kwargs):
        super().__init__(**kwargs)
        self.common_feats = common_feats

    def partial_fit(self, X, y=None):
        super().partial_fit(X, y)

        if self.common_feats:
            for group in self.common_feats:
                # calculate min and max across all features within a group, instead of calculating for individual features
                group = np.array(group)
                self.data_min_[group] = np.min(self.data_min_[group])
                self.data_max_[group] = np.max(self.data_max_[group])
                self.data_range_[group] = self.data_max_[group] - self.data_min_[group]

                self.scale_[group] = ((self.feature_range[1] - self.feature_range[0]) / CustomMinMaxScaler._handle_zeros_in_scale(self.data_range_[group]))
                self.min_[group] = self.feature_range[0] - self.data_min_[group] * self.scale_[group]

        return self

Could someone please let me know if this is correct?

[glemaitre]

Write a couple of unit tests to ensure that you have the expected behaviour.

[Genius1237]

Thank you for your suggestion. This is something that I could do, but the problem is that I am not aware of what corner cases I should be checking for.

For example, in the partial_fit implementation of the original class, there is some code associated with repeated calls to the fit and partial_fit method. I believe that these functions will be called only once (and hence I don't need to worry about this), but I am not 100% sure.

Is there some place that documents the ways in which the different functions of the class will be called?

[ogrisel]

partial_fit is meant to be called several times to incrementally process data by chunks, typically when the full dataset does not fit in memory all at once.

fit is meant to be called only once with the full dataset and forget anything from previous calls.

[ogrisel]

I think this would be an interesting option to add to the sklearn.preprocessing.MinMaxScaler class. It could be very useful to preprocess homogeneous numerical features (e.g. pixel brightness varlues or measures by arrays of sensors that all measure the same physical units) without breaking cross-feature homogeneity.

Feel free to open a pull request if you are interested, modifying the existing MinMaxScaler class introducing a new constructor parameter (homogeneous=False for instance) instead of defining a new subclass. This would require writing tests as @glemaitre suggested.

[ogrisel]

Also if we implement the homogeneous variants of sklearn.preprocessing.MinMaxScaler then we can use ColumnTransformers to define groups of homogeneous features and pass them each a dedicated instead of MinMaxScaler(homogeneous=True).

[Genius1237]

I will try to get this done when I get some free time. I am guessing that if this is implemented, using the ColumnTransformers method will eventually meet my needs of having a mix of homogeneous and non-homogeneous features.

My knowledge of writing unit tests for python are a bit rusty. Is there someplace within the sklearn docs that I could look at?

[NicolasHug]

We have pytest specific tips at https://scikit-learn.org/stable/developers/tips.html#useful-pytest-aliases-and-flags

But for starters you can just look at some of the current tests we have and follow the pytest getting started guide

[bharatr21]

Can I work on extending this module for homogeneous features if @Genius1237 is okay? Always curious to explore what feature preprocessing is doing under the hood, and this is a good opportunity!

[Genius1237]

Go ahead