Skip to content

claudiom4sir/VideoRestoration

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 

Repository files navigation

Video restoration based on deep learning: a comprehensive survey

Claudio Rota, Marco Buzzelli, Simone Bianco, Raimondo Schettini

Artificial Intelligence Review, Springer Nature, 2022

[Open Access PDF]

Abstract

Video restoration concerns the recovery of a clean video sequence starting from its degraded version. Different video restoration tasks exist, including denoising, deblurring, super-resolution, and reduction of compression artifacts. In this paper, we provide a comprehensive review of the main features of existing video restoration methods based on deep learning. We focus our attention on the main architectural components, strategies for motion handling, and loss functions. We analyze the standard benchmark datasets and use them to summarize the performance of video restoration methods, both in terms of effectiveness and efficiency. In conclusion, the main challenges and future research directions in video restoration using deep learning are highlighted.

Citation

@article{rota2023video,
  title={Video restoration based on deep learning: a comprehensive survey},
  author={Rota, Claudio and Buzzelli, Marco and Bianco, Simone and Schettini, Raimondo},
  journal={Artificial Intelligence Review},
  volume={56},
  number={6},
  pages={5317--5364},
  year={2023},
  publisher={Springer}
}

Methods for video restoration

Name Task(s) Paper Code
VESPCN Super-resolution Real-time video super-resolution with spatio-temporal networks and motion compensation (CVPR 2017) [Pytorch]
DBN Deblurring Deep Video Deblurring for Hand-held Cameras (CVPR 2017) [Torch]
STRCNN Deblurring Online video deblurring via dynamic temporal blending network (ICCV 2017)
DBLRGAN Deblurring Adversarial spatio-temporal learning for video deblurring (TIP 2018)
DUF Super-resolution Deep Video Super-Resolution Network Using Dynamic Upsampling FiltersWithout Explicit Motion Compensation (CVPR 2018) [TensorFlow]
MFQE Compression artifact reduction Multi-Frame Quality Enhancement for Compressed Video (CVPR 2018) [TensorFlow]
ToFlow Denoising, Super-resolution, Compression artifact reduction Video Enhancement with Task-Oriented Flow (IJCV 2019) [Torch]
DVDNet Denoising Dvdnet: A fast network for deep video denoising (ICIP 2019) [Pytorch]
MFQE2.0 Compression artifact reduction MFQE 2.0: A New Approach for Multi-frame Quality Enhancement on Compressed Video (TPAMI 2019) [TensorFlow]
STFAN Deblurring Spatio-Temporal Filter Adaptive Network for Video Deblurring (ICCV 2019) [Pytorch]
EDVR Deblurring, Super-resolution Edvr: Video restoration with enhanced deformable convolutional networks (CVPR 2019) [Pytorch]
PFNL Super-resolution Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations (ICCV 2019) [Pytorch/TensorFlow]
ViDeNN Denoising ViDeNN: Deep Blind Video Denoising (CVPR 2019) [TensorFlow]
IFI-RNN Deblurring Recurrent neural networks with intra-frame iterations for video deblurring (CVPR 2019)
FITVNet Denoising First image then video: A two-stage network for spatiotemporal video denoising (2019)
VNLNet Denoising A Non-Local CNN for Video Denoising (ICIP 2019)
NL-ConvLSTM Compression artifact reduction Non-local convlstm for video compression artifact reduction (ICCV 2019) [Pytorch]
MB2D Deblurring Blur more to deblur better: Multi-blur2deblur for efficient video deblurring (2020)
FastDVDNet Denoising Fastdvdnet: Towards real-time deep video denoising without flow estimation (CVPR 2020) [Pytorch]
TDAN Super-resolution Tdan: Temporally-deformable alignment network for video super-resolution (CVPR 2020) [Pytorch]
ESTRNN Deblurring Efficient spatio-temporal recurrent neural network for video deblurring (ECCV 2020) [Pytorch]
STDF Compression artifact reduction Spatio-temporal deformable convolution for compressed video quality enhancement (AAAI 2020) [Pytorch]
MuCAN Super-resolution Mucan: Multi-correspondence aggregation network for video super-resolution (ECCV 2020) [Pytorch]
RViDeNet Denoising Supervised raw video denoising with a benchmark dataset on dynamic scenes (CVPR 2020) [Pytorch]
RSDN Super-resolution Video super-resolution with recurrent structure-detail network (ECCV 2020) [Pytorch]
CDVD-TSP Deblurring Cascaded deep video deblurring using temporal sharpness prior (CVPR 2020) [Pytorch]
Evrnet Denoising, Super-resolution, Compression artifact reduction Evrnet: Efficient video restoration on edge devices (ICM 2021)
MMNet Denoising Multiframe-to-multiframe network for video denoising (TOM 2021)
RFDA Compression artifact reduction Recursive fusion and deformable spatiotemporal attention for video compression artifact reduction (ICM 2021) [Pytorch]
PVDNet Deblurring Recurrent video deblurring with blur-invariant motion estimation and pixel volumes (TOG 2021) [Pytorch]
MaskDNGAN Denoising Multi-Stage Raw Video Denoising with Adversarial Loss and Gradient Mask (ICCP 2021) [Pytorch]
PaCNet Denoising Patch craft: Video denoising by deep modeling and patch matching (ICCV 2021) [Pytorch]
BasicVSR Super-resolution BasicVSR: The search for essential components in video super-resolution and beyond (CVPR 2021) [Pytorch]
BasicVSR++ Super-resolution BasicVSR++: Improving video super-resolution with enhanced propagation and alignment (CVPR 2022) [Pytorch]

Datasets for video restoration

Name Task(s) Usage Repository
GOPRO Deblurring Train/Test [Link]
DVD Deblurring Train/Test [Link]
BSD Deblurring Train/Test [Link]
REDS Deblurring, Super-resolution Train/Test [Link]
Vid4 Super-resolution Test [Link]
UDM10 Super-resolution Test [Link]
SPMCS Super-resolution Test [Link]
Vimeo90K Denoising, Super-resolution, Compression artifact reduction Train/Test [Link]
MFQEv2 Compression artifact reduction Train/Test [Link]
CRVD Denoising Train/Test [Link]
Set8 Denoising Test [Link]
DAVIS 2017 Denoising Train/Test [Link]

Contacts

If you have any question, please contact Claudio Rota at c.rota30@campus.unimib.it