Restore From Restored:
Video Restoration With Pseudo Clean Video

[Paper] [Supplementary]

Abstract

In this study, we propose a self-supervised video denoising method called "restore-from-restored." This method fine-tunes a pre-trained network by using a pseudo clean video during the test phase. The pseudo clean video is obtained by applying a noisy video to the baseline network. By adopting a fully convolutional neural network (FCN) as the baseline, we can improve video denoising performance without accurate optical flow estimation and registration steps, in contrast to many conventional video restoration methods, due to the translation equivariant property of the FCN. Specifically, the proposed method can take advantage of plentiful similar patches existing across multiple consecutive frames (i.e., patch-recurrence); these patches can boost the performance of the baseline network by a large margin. We analyze the restoration performance of the fine-tuned video denoising networks with the proposed self-supervision-based learning algorithm, and demonstrate that the FCN can utilize recurring patches without requiring accurate registration among adjacent frames. In our experiments, we apply the proposed method to state-of-the-art denoisers and show that our fine-tuned networks achieve a considerable improvement in denoising performance.

Environments

• Python 3.7
• Pytorch 1.7.1

Test

Examples of demo codes

Arguments

--video_dir: Path of test video frames
--sigma: Gaussian noise level, one of [15, 25, 40]
--online: Run RFR in online mode
--offline: Run RFR in offline mode

Baseline denoising networks

1. DnCNN

• Only for grayscale video frames
• We provide a single video in this repository (i.e., factory_1080p30)

python run_rfr.py --gpu 0 --net dncnn --video_dir ./testsets/derf_HD/factory_1080p30 --sigma 25 --online

2. VNLnet and FastDVDnet

• Download dataset (RGB) used for experiments in our paper (Derf and Davis)

python run_rfr.py --gpu 0 --net fastdvdnet --video_dir [PATH_OF_VIDOE_FRAMES] --sigma 25 --online --offline

PYOPENCL_CTX='0' python run_rfr.py --gpu 0 --net vnlnet --video_dir [PATH_OF_VIDOE_FRAMES] --sigma 25 --online --offline

Citation

@InProceedings{Lee_2021_CVPR,
    author    = {Lee, Seunghwan and Cho, Donghyeon and Kim, Jiwon and Kim, Tae Hyun},
    title     = {Restore From Restored: Video Restoration With Pseudo Clean Video},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2021},
    pages     = {3537-3546}
}