JPEG compression

[rgeirhos]

Javier Portilla, Marina Martinez and Emilio Sansano reached out since they were unable to exactly reproduce the experimental stimuli for blurred gray-level 224x224 images starting from the originals in colors. I'm pasting their observations - based on extensive analysis - here in case anyone else might come across this in the future. As a TL/DR: it's likely that the stimuli were saved to JPEG as an intermediate step.

"After a deep and exhaustive search for the source of the problem, we have concluded that it must be due to an unwilling jpeg compression somewhere in your pipeline from the imagenet color originals of different sizes and aspect ratios to the reference 224x224 gray level images. We have reached that conclusion by observing

1. Statistical differences between the image we obtain following your description of the pipeline preparation and the actual reference image in your repository (range normalised to [0,1]):
   max average diff: 0.03326154666967139
   min average diff: 0.01042876110608175
   average MSE: 0.001010270576355128
   average PSNR: 31.142782442676072
   This PSNR is compatible with a moderate JPEG compression level.

2. Visual comparisons
   

For the image conversion pipeline, from the imagenet original color images to the reference ones, we have tried all possible combinations of order and type of cropping, scaling and RGB2gray transformations, and the results above are the best we get. Clearly, the problem is not due to a wrong scale/crop/rgb2gray order, nor to the antialiasing kernel for scaling or to using different RGB2gray functions (or an eventual gamma correction). It must be an accidental JPEG compression at some intermediate step of the image pipeline.

The good news:
We do not think this problem compromise your research results and conclusions, as (1) blur wipes out most of the jpeg-error (e.g., PSNR increases to more than 51 dB when blurring images with condition sigma=10), and (2) under the observation conditions of the psychophysical tests you have carried out (each observed image spanning a 3x3 degree visual angle) differences seems too small to have an impact on classification. Similarly, they should not have a significant impact on ANN's classification. Thus, your comparison between human and ANN's performance and its conclusions must be valid yet."

Many thanks to Javier, Marina and Emilio for investigating and reporting this!