Problems with ForecastingGridSearchCV using WindowSummarizer

[acartro]

Good morning,

I am using sktime version 0.12.1.

I have been trying to perform a grid search for a random forest using WindowSummarizer for lag creation.
When I do the search for the best parameters, it doesn't let me do it. Does anyone know why this is happening? What am I doing wrong?

Thanks

import sktime
print(sktime.__version__)

import numpy as np
import pandas as pd
from sktime.datasets import load_airline
from sktime.forecasting.model_selection import temporal_train_test_split
from sktime.transformations.series.date import DateTimeFeatures
from sktime.transformations.series.summarize import WindowSummarizer
from sktime.forecasting.trend import PolynomialTrendForecaster
from sktime.transformations.series.detrend import Detrender
from sktime.forecasting.compose import TransformedTargetForecaster
from sktime.transformations.series.detrend import Deseasonalizer
from sklearn.ensemble import RandomForestRegressor
from sktime.forecasting.compose import ForecastingPipeline, make_reduction
from sktime.forecasting.model_selection import ExpandingWindowSplitter
from sktime.forecasting.model_selection import ForecastingGridSearchCV

# load dataset
y = load_airline()
# train test split dataset
y_train, y_test = temporal_train_test_split(y)

# Forecasting horizon
FH = np.arange(1, 12+1)

# parameters to create lag features
kwargs = {
    "lag_feature": {
        "lag": [1],
        "mean": [[1, 3], [3, 6]],
        "std": [[1, 4]]
    }
}

# pipeline steps to create datatime features
steps = [
    ("daily_season", DateTimeFeatures(ts_freq="M")),
    ("daily_season2", DateTimeFeatures(
        manual_selection=["quarter_of_year"])),
]

# transform target with window summarizer
forecaster = TransformedTargetForecaster(
    [
        ("deseasonalize", Deseasonalizer(model="multiplicative", sp=12)),
        ("detrend", Detrender(forecaster=PolynomialTrendForecaster(degree=1))),
        (
            "forecast",
            make_reduction(
                RandomForestRegressor(random_state=1),
                scitype="tabular-regressor",
                transformers=[WindowSummarizer(
                    **kwargs, n_jobs=1, truncate="bfill")],
                window_length=None,
                strategy="recursive",
            ),
        ),
    ]
)


# Pipeline global forecaster
pipe_global = ForecastingPipeline(
    steps=steps + [("forecaster", forecaster)])

# Grid search on random forest regressor
param_grid = {"forecaster__forecast__max_depth": [2, 3, 4]}
cv = ExpandingWindowSplitter(
                 initial_window=int(len(y_train) * 0.7))
gscv = ForecastingGridSearchCV(
    pipe_global, strategy="refit", cv=cv, param_grid=param_grid
)

# fit grid search
gscv.fit(y_train)

[fkiraly]

I think your problem is that the parameter string to access the max_depth of the random forest is incorrect.

What you want is to access the RandomForestRegressor, so you need the estimator within the reducer, which is the forecast component inside the TransformedTargetForecaster, which is the forecaster component inside the ForecastingPipeline, which is the forecaster inside the ForecastingGridSearch.

So, the grid search parameter should be called, I think, "forecaster__forecast__estimator__max_depth", and not "forecaster__forecast__max_depth".
The error comes from the forecast component not having a max_depth - which is correct, since it is not RandomForestRegressor but RecursiveTabularRegressionForecaster.

[fkiraly]

... there's a lot of nesting, so I wonder whether one could make the user experience cleaner here. E.g., should set_params not just recognize, say, max_depth instead of requiring forecaster__forecast__estimator__max_depth, if there is only one parameter of that name, max_depth?

That would be a little departure from scikit-learn, but it might be less ... fiddly?

[fkiraly]

PS: what is helpful in these cases is calling get_params(), that lists all valid parameter strings as keys.

[fkiraly]

prototype of "just accept max_depth if there is only one instance in the parameter tree, please" here: #2931

[fkiraly]

Shortified verion of your code (requires the PR and dev version).

Would appreciate feedback whether this is "nicer" or not.

import numpy as np
import pandas as pd
from sktime.datasets import load_airline
from sktime.forecasting.all import *
from sktime.transformations.all import *
from sklearn.ensemble import RandomForestRegressor

# load dataset
y = load_airline()
# train test split dataset
y_train, y_test = temporal_train_test_split(y)

# Forecasting horizon
FH = np.arange(1, 12+1)

# parameters to create lag features
kwargs = {
    "lag_feature": {
        "lag": [1],
        "mean": [[1, 3], [3, 6]],
        "std": [[1, 4]]
    }
}

# pipeline steps to create datatime features
steps = (
    DateTimeFeatures(ts_freq="M")
    * DateTimeFeatures(manual_selection=["quarter_of_year"])
)

# transform target with window summarizer
forecaster = (
    Deseasonalizer(model="multiplicative", sp=12)
    * Detrender(forecaster=PolynomialTrendForecaster(degree=1))
    * make_reduction(
          RandomForestRegressor(random_state=1),
          scitype="tabular-regressor",
          transformers=[WindowSummarizer(
              **kwargs, n_jobs=1, truncate="bfill")],
          window_length=None,
          strategy="recursive",
    ),
)


# Pipeline global forecaster
pipe_global = ForecastingPipeline(
    steps=steps + [("forecaster", forecaster)])

# Grid search on random forest regressor
param_grid = {"max_depth": [2, 3, 4]}

cv = ExpandingWindowSplitter(
                 initial_window=int(len(y_train) * 0.7))
gscv = ForecastingGridSearchCV(
    pipe_global, strategy="refit", cv=cv, param_grid=param_grid
)

# fit grid search
gscv.fit(y_train)

[NasreddineD]

Hi @fkiraly , I copied your code above and got the following error.

TypeError                                 Traceback (most recent call last)
Cell In[27], line 48
     32 forecaster = (
     33     Deseasonalizer(model="multiplicative", sp=12)
     34     * Detrender(forecaster=PolynomialTrendForecaster(degree=1))
   (...)
     42     ),
     43 )
     46 # Pipeline global forecaster
     47 pipe_global = ForecastingPipeline(
---> 48     steps=steps + [("forecaster", forecaster)])
     50 # Grid search on random forest regressor
     51 param_grid = {"max_depth": [2, 3, 4]}

TypeError: unsupported operand type(s) for +: 'TransformerPipeline' and 'list'

[fkiraly]

which sktime version are you using? Possibly an old one?

[NasreddineD]

@fkiraly, I first tried it with 0.16.1, and now with 0.18 and still the same error.

[acartro]

I didn't know about the existence of get_params().
In my opinion, making a cleaner user experience is always welcome. But I'm fine with the current way of declaring.

By the way, modifying the parameter to forecaster__forecast__estimator__max_depth worked for me.

Thank you very much.

[fkiraly]

I didn't know about the existence of get_params().

Ah, I see - get_params is actually a scikit-learn interface point, and sktime is compatible with that.
So, the same applies for anything you do in scikit-learn.

Perhaps we should also showcase the scikit-learn native interface points in the sktime tutorials, just to tell users that they are there.