How can I perform multiple transformations of columns with some columns being same across the transformations

[tejavenkatk]

For my usecase, I wanted to perform target encoding for some columns (say c1, c2,c3) and I also want to perform imputation for a column (c4) and I now wanted to perform standardscalar (once the previous target encoding and imputation are performed) for all these and more columns (c1,c2,c3,c4,c5,c6,c7,c8).

Note I also have more columns that I need to perform, say OHE, but those columns are disjoint from the above c1...c8.

I am not sure how to use column transformer and pipeline, such that when I perform different transformations with some of the columns being same among the transformations (as in the above example). Is there a way to do this as part of the pipeline. Thank you.

[glemaitre]

Moving this into a discussion

[glemaitre]

In the ColumnTransformer, you can use the columns more than one. You can also use a complex pipeline to transform the data.

You could also use a pipeline with 2 ColumnTransformers, one after the other if needed. In this last case, tracking the column name-indices is not easy due to the reordering but this is feasible if really you want to.

preprocessor_1 = make_column_transformer(
  ("target_encoder", TargetEncoder(), ["c1", "c2", "c3"]),
  ("imputer", SimpleImputer(), ["c4"]),
  ("others", "passthrough", [f"c{i}" for i in range(5, 9)],
)
model = make_pipeline(
  preprocessor_1, StandardScaler(), Predictor()
)

So if you would like to add `ColumnTransformer` instead of only a `StandardScaler`, this is where you would need to provide the indices of the reordered columns because the columns names have been dropped then.

[tejavenkatk]

Thank you. A couple of questions.

In option #1 - where we can use columns more than once in column transformer : In this case the transformations are parallel applied to the same column ( if it is listed for two transformations) and not serially as we expect. For example, if we would want to do imputation and then standardscalar on the same column.

In option #2 - using 2 column transformers in a pipeline : The first column transformer will not return the column names right ? In this case, how should I specify the column names for the second column transformer, should I be figuring out the order of the columns and specify the indices in this case ? Other challenge in this case is if we use one hot encoding in the the first column transformer with multiple columns (where it would be difficult to get the indices - although feasible in some cases where the categories are fixed and limited).

[glemaitre]

In option #1

Imputation + standardization should be in a sequence, so you can use a pipeline as a transformer (as below). What I meant by reusing column, is that you can specify the same column name several time as below:

preprocessor_1 = make_column_transformer(
  ("whatever", make_pipeline(SimpleImputer(), StandardScaler()), ["c4"]),
  ("repeat", make_pipeline(SimpleImputer(), QuantileTransformer()), ["c4"]),
)

The transformation `"whatever"` and `"repeat`" will be done in parallel because they are independent. The imputation + scaling will be done as a sequence because it is in a pipeline.

> The first column transformer will not return the column names right ? In this case, how should I specify the column names for the second column transformer, should I be figuring out the order of the columns and specify the indices in this case ? 

Yes this is the tricky part. You can infer the order by using `get_feature_names_out` method that will return the column names at any stage of the pipeline. However, we don't yet support dataframe in/out (it is on its way). So one can know about feature names with:

```python
# Remove the predictor to know the feature names after the preprocessing done
model[:-1].get_feature_names_out()

and not note you can slice model at any step that you want to know the intermediate column names.

[NeoAcereo]

Thank you. I needed the answer to the first question.