SciTab

The project page for "SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables

What's New

[2023.11.12] The EMNLP poster and slides are ready! Please check the doc folder.

[2023.10.22] The camera ready version is ready! Please check the doc folder.

[2023.10.12] More training materials, including the recruitment advertisement, registration form, and the agreement sheet have been uploaded.

[2023.10.08] The SCITAB work has been accepted at EMNLP 2023 main conference! Stay tuned for the camera-ready version!

[2023.05.20] The project page has been built!

Introduction

This repository contains the code and data for the paper SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables.

Dataset

The dataset is stored as json files in folder "dataset", each entry has the following format:

"paper": the paper name
"paper_id": the paper id
"table_cpation": the table caption
"table_column_names": the table column names
"table_content_values": the table content
"id": the unique claim id
"claim": the claim texts
"label": the label of the claim, one of the labels from {supports, refutes, not enough info}
"table_id": the unique table id   

Citation

If you find this project useful, please cite it using the following format:

@inproceedings{Luscitab23,
  author       = {Xinyuan Lu and
                  Liangming Pan and
                  Qian Liu and
                  Preslav Nakov and
                  Min{-}Yen Kan},
  title        = {{SCITAB:} {A} Challenging Benchmark for Compositional Reasoning and
                  Claim Verification on Scientific Tables},
  booktitle    = {Proceedings of the 2023 Conference on Empirical Methods in Natural
                  Language Processing, {EMNLP} 2023, Singapore, December 6-10, 2023},
  pages        = {7787--7813},
  publisher    = {Association for Computational Linguistics},
  year         = {2023},
  url          = {https://aclanthology.org/2023.emnlp-main.483}
}