NTIRE2021-IQA-MACS (tensorflow 2)

Pytorch Version : https://github.com/AliRoyat/NTIRE2021-IQA-MACS-Pytorch

Evaluation

1. Download pretrained models from here: [1] Trained Models (tensorflow 2) (~135 MB)
   
2. Extract the models.zip file and put the models in ./models/.

Evaluation on a single image

1. Run evaluation_single_image.py

   python3 evaluation_single_image.py --ref ./test_images/ref.bmp --distorted ./test_images/dist.bmp
   

The output is:

  -------------------------------------
  Image Quality Score:  1381.0543870192307

Evaluation on NTIRE images

1. Set the directories of the validation reference images and distorted images in evaluation_ntire.py

2. Run evaluation_ntire.py

   python3 evaluation_ntire.py
   

The output scores will be logged in output.txt.

Training Networks From Scratch

Preparing Datasets

Download datasets:
[1] TID 2008 dataset (~550 MB)
[2] TID 2013 dataset (~908 MB)
[3] PieAPP (~2.2 GB)
[4] PIPAL dataset

Please Make all the formats in the reference images of TID datasets in Upper Case (.BMP) (Typically I25 in the reference images has .bmp format instead of .BMP)
Set datasets directories in create_data_pieapp_tid.py and run it:

  python3 prepare_dataset/create_data_pieapp_tid.py

Set PIPAL dataset directory in create_data_pipal.py and run it: (Needs at least 16 GB RAM)

  python3 prepare_dataset/create_data_pipal.py

We can not directly include spearman loss function to train the network, spearman's rank correlation coefficient formula is:

It is a non-differentiable function because of the ranking operation. Instead, we train a network to learn sorting the inputs and include this surrogate metric in the loss function:

Training Surrogate Ranking Model (Surrogate Spearman Loss Model)

Set total number of training data in train.py and run it: (Default 2M data)

  python3 ranking_model/train.py

(Optional) Now, you can run test.py to see the diffrence in the surrogate spearman and the true spearman for some generated data:

  python3 ranking_model/test.py

Note that, the model input and output for the surrogate ranking model is 16, which is same as the training batch size of the IQA model.

Training Models (including pretraining models on PieAPP and TID datasets)

1. Use the configs text files to train the models using train_simple.py for the basic model architecture, train_bn.py for the model architecture with batch normalization, train_attention.py for the model architecture with attention mechanism and residual blocks and train_tiled.py for model architecture which splits the image into square tiles and pass them to a ConvLSTM2D layer. The model architectures are shown in the following figures: Basic Model architecture: Model architecture with attention layer and residual blocks: Model architecture with tiling the input and passing it to a ConvLSTM2D:

Citation

Kindly cite our paper upon using this repository:

  @misc{ayyoubzadeh2021asna,
  title={(ASNA) An Attention-based Siamese-Difference Neural Network with Surrogate Ranking Loss function for Perceptual Image Quality Assessment}, 
  author={Seyed Mehdi Ayyoubzadeh and Ali Royat},
  year={2021},
  eprint={2105.02531},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
   }

References

[1] N. Ponomarenko, V. Lukin, A. Zelensky, K. Egiazarian, M. Carli, F. Battisti, "TID2008 - A Database for Evaluation of Full-Reference Visual Quality Assessment Metrics", Advances of Modern Radioelectronics, Vol. 10, pp. 30-45, 2009.
[2] N. Ponomarenko, L. Jin, O. Ieremeiev, V. Lukin, K. Egiazarian, J. Astola, B. Vozel, K. Chehdi, M. Carli, F. Battisti, C.-C. Jay Kuo, Image database TID2013: Peculiarities, results and perspectives, Signal Processing: Image Communication, vol. 30, Jan. 2015, pp. 57-77.
[3] PieAPP: Perceptual Image-Error Assessment Through Pairwise Preference, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[4] PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration , Jinjin Gu, Haoming Cai, Haoyu Chen, Xiaoxing Ye, Jimmy Ren, Chao Dong ECCV 2020