Releases: ahmedmagdiosman/clevr-xai
v1.0
Public Release of CLEVR-XAI v1.0
CLEVR-XAI was introduced in the paper:
Title: CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations
Journal: Information Fusion, Volume 81, May 2022, Pages 14-40
Authors: Leila Arras*, Ahmed Osman*, Wojciech Samek (*equal contribution)
Link: https://doi.org/10.1016/j.inffus.2021.11.008
Preprint: https://arxiv.org/abs/2003.07258
(This publication was submitted: January 2021, accepted & available online: November 2021)
The CLEVR-XAI dataset is organized into the following structure:
Note: due to Github's file size limitations, the folders images/ and masks/ were uploaded separately. Please move them to the same root directory after downloading and extracting the ZIP files.
./
├── CLEVR-XAI_scenes.json # CLEVR-XAI scenes, one scene per image [10,000 scenes in total]
├── CLEVR-XAI_simple_questions.json # CLEVR-XAI-simple questions [39,761 questions in total]
├── CLEVR-XAI_complex_questions.json # CLEVR-XAI-complex questions [100,000 questions in total]
├── images/ # CLEVR-XAI images
│ ├── CLEVR_unique_000000.png
│ ├── CLEVR_unique_000001.png
│ ├── ... [10,000 images in total]
├── masks/ # CLEVR-XAI segmentation masks
│ ├── CLEVR_unique_000000.png
│ ├── CLEVR_unique_000001.png
│ ├── ... [10,000 masks in total]
├── ground_truth_simple_questions_single_object/ # GT Single Object, on CLEVR-XAI-simple questions
│ ├── 0.npy
│ ├── 1.npy
│ ├── ... [39,761 ground truths in total]
├── ground_truth_simple_questions_all_objects/ # GT All Objects, on CLEVR-XAI-simple questions
│ ├── 0.npy
│ ├── 1.npy
│ ├── ... [39,761 ground truths in total]
├── ground_truth_complex_questions_unique/ # GT Unique, on CLEVR-XAI-complex questions
│ ├── 0.npy
│ ├── 1.npy
│ ├── ... [89,873 ground truths in total]
│ └── stats.json
├── ground_truth_complex_questions_unique_firstnonempty/ # GT Unique First-non-empty, on CLEVR-XAI-complex questions
│ ├── 0.npy
│ ├── 1.npy
│ ├── ... [99,786 ground truths in total]
│ └── stats.json
├── ground_truth_complex_questions_union/ # GT Union, on CLEVR-XAI-complex questions
│ ├── 0.npy
│ ├── 1.npy
│ ├── ... [99,786 ground truths in total]
│ └── stats.json
└── ground_truth_complex_questions_all_objects/ # GT All Objects, on CLEVR-XAI-complex questions
├── 0.npy
├── 1.npy
├── ... [100,000 ground truths in total]
└── stats.json
Description:
The top three JSON files contain the CLEVR-XAI scenes and simple, resp. complex, questions in the same format as in the original CLEVR dataset (see CLEVR paper: https://arxiv.org/pdf/1612.06890.pdf, CLEVR release page: https://cs.stanford.edu/people/jcjohns/clevr/, and the CLEVR v1.0 dataset README).
A few minor differences w.r.t. the original CLEVR v1.0 dataset are:
- In the JSON files, we introduced a new value for the field named "split" defining the dataset's split. In the original CLEVR dataset the value of this field can be one of the following strings: "train", "val" or "test". In our CLEVR-XAI dataset this value is "unique". This has historical reasons, as initially we were considering only "unique" objects as ground truths (i.e., the output of the basic functions of type "unique" in the question functional program), accordingly the file names in the folders images/ and masks/ have the form CLEVR_unique_<image_index>.png (instead of CLEVR_train_<image_index>.png, CLEVR_val_<image_index>.png and CLEVR_test_<image_index>.png as in the original CLEVR dataset).
- In the questions' JSON files, in order to access the type of the basic functions within the functional program of each question the keyword 'function' which was used in the original CLEVR dataset was replaced by 'type' in CLEVR-XAI.
- In the complex questions' JSON file, for questions having a boolean as answer, the 'yes'/'no' answer in the original CLEVR dataset was replaced by the boolean True/False in CLEVR-XAI.
The PNG files inside the folder images/ contain the CLEVR-XAI images, in the same fashion as in the original CLEVR dataset, of size 480 x 320.
The PNG files inside the folder masks/ contain the CLEVR-XAI segmentation masks (one per image/scene), of size 480 x 320, with a unique color assigned to each object in the scene (the color's values can be found in the scenes' JSON file under the keyword 'mask_color'). These masks served as a basis for creating our ground truths. They are provided for completeness reasons and to allow further data processing, note though that in a standard use-case following the evaluation approach we introduced in our publication you don't need those masks.
The ground truths are given as Numpy files, within a separate folder for each ground truth type. Each Numpy file's name has the form <question_index>.npy. It contains a Numpy ndarray of data type boolean, and of shape 128 x 128. This shape is equal to the shape of the input images of the Relation Network model (see Relation Network paper: https://arxiv.org/pdf/1706.01427.pdf), which was used in our evaluation framework (see our publication for more details) to evaluate and compare XAI methods.
Additionally, for the CLEVR-XAI-complex ground truths, we provide for each type of ground truth a JSON file named stats.json which contains the number of objects per ground truth and the total number of objects in the scene, in the format: {'<question_index>': {'target_objects': int, 'total_objects': int}}.
Citation:
@article{ARRAS_etal2022,
title = {CLEVR-XAI: A benchmark dataset for the ground truth evaluation of neural network explanations},
journal = {Information Fusion},
volume = {81},
pages = {14-40},
year = {2022},
doi = {https://doi.org/10.1016/j.inffus.2021.11.008},
author = {Leila Arras and Ahmed Osman and Wojciech Samek}
}
First public release of simply-CLEVR
The simply-CLEVR dataset was introduced in the paper arXiv:2003.07258v1 (which was released on March 2020).
Please read the README in the branch simply-clevr of the repo for more information.
Content:
- The images, questions, and objects segmentation masks are in simply-clevr-10k.zip.
simply-clevr-10k_ground_truth.zipcontains the single object ground truths.simply-clevr-10k_ground_truth_target_all.zipcontains the all-object ground truths.blender_clevr.sifis the pre-built singularity image for image generation.eval-unique-clevr.sifis the pre-built singularity image for ground truth mask generation. Not currently used for relevance evaluation. Please refer to this gist.