CPL: Continual Predictive Learning from Videos (CVPR 2022 Oral)

A PyTorch implementation of our paper: Continual Predictive Learning from Videos.

Introduction

In this [paper], we study a new continual learning problem in the context of video prediction, and observe that most existing methods suffer from severe catastrophic forgetting in this setup. To tackle this problem, we propose the continual predictive learning (CPL) approach, which learns a mixture world model via predictive experience replay and performs test-time adaptation with non-parametric task inference. Our approach is shown to effectively mitigate forgetting and remarkably outperform the naïve combinations of previous art in video prediction and continual learning.

Get Started

1. Install Python 3.8, PyTorch 1.9.0 for the code.

2. Download data. KTH action dataset

3. Train the CPL model. You can use train.sh/test.sh to train/test the CPL model. The learned model will be saved in the --save_dir folder. The generated future frames will be saved in the --gen_frm_dir folder.

cd script/
sh train.sh

4. To train the base model, use the following script.

cd script/
sh train_base.sh

Follow-up Work

We extended CPL for continual model-based RL. [paper] [code]

Acknowledgement

We appreciate the following github repos where we borrow code from:

https://github.com/thuml/predrnn-pytorch

https://github.com/AntixK/PyTorch-VAE