Pretrain on forecasting/imputation and finetune on classification

[Glaadiss]

I'm working with a timeseries dataset full of sensor data, but only a small portion is labeled. My plan is to pretrain the model on the entire dataset and then fine-tune it using the labeled subset. This dataset is rich in features and spans numerous devices.

My goal is to assign the correct label (8 possible classes) for every minute, so this is either timeseries clustering/segmentation task or a classification task for every minute.

For every minute I have 60 records (one for each second), and every record includes a few values such as mean, std, min, max calculated based on 100Hz data for a given second.

I'm considering using TimesNet for forecasting and/or imputation during pretraining phase, and classification during fine-tuning phase. Has anyone here experimented with TimesNet for similar applications? I'd love to hear any insights or advice you might have. If it hasn't been done, I'm eager to explore and contribute to this area. Any tips or recommendations would be greatly appreciated!

[wuhaixu2016]

Hi, great question.
I would like to refer to our paper on time series pre-training: SimMTM: https://github.com/thuml/SimMTM

[Glaadiss]

Hi @wuhaixu2016 Thank you for the link to SimMTM - I'll check the details! However, I noticed the pertaining and fine-tuning are done on the classification task. I was wondering if anyone tried doing pretraining on the forecasting/imputation task and reusing weights for the final classification task. Unfortunately, I don't have any discrete labels that I can use for pretraining.