FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations (NAACL 2022)

This repository contains the code for the paper "FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations".

FactGraph is an adapter-based method for assessing factuality that decomposes the document and the summary into structured meaning representations (MR):

In FactGraph, summary and document graphs are encoded by a graph encoder with structure-aware adapters, along with text representations using an adapter-based text encoder. Text and graph encoders use the same pretrained model and only the adapters are trained:

Environment

The easiest way to proceed is to create a conda environment:

conda create -n factgraph python=3.7
conda activate factgraph

Further, install PyTorch and PyTorch Geometric:

pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install torch-scatter==2.0.9 -f https://data.pyg.org/whl/torch-1.9.0+cu111.html
pip install torch-sparse==0.6.12 -f https://data.pyg.org/whl/torch-1.9.0+cu111.html
pip install torch-geometric==2.0.3

Install the packages required:

pip install -r requirements.txt

Finally, create the environment for AMR preprocessing:

cd data/preprocess
./create_envs_preprocess.sh
cd ../../

FactCollect Dataset

FactCollect is created consolidating the following datasets:

Dataset	Datapoints
Wang et al. (2020)	953	Link
Kryscinski et al. (2020)	1434	Link
Maynez et al. (2020)	2500	Link
Pagnoni et al. (2021)	4942	Link

• FactCollect uses two datasets released under licenses.
  • FactCC is under BSD-3. Copyright (c) 2019, Salesforce.com, Inc. All rights reserved.
  • XSum Hallucinations is under CC BY 4.0.

For generating FactCollect dataset, execute:

conda activate factgraph
cd data
./create_dataset.sh
cd ..

Running trained FactGraph Models

First, download FactGraph trained checkpoints:

cd src
./download_trained_models.sh

To run FactGraph:

./evaluate.sh factgraph <file> <gpu_id>

To run FactGraph edge-level:

./evaluate.sh factgraph-edge <file> <gpu_id>

<file> is a JSON line file with the following format:

{'summary': summary1, 'article': article1}
{'summary': summary2, 'article': article2}
...

where 'summary' is a single sentence summary.

Training FactGraph

Preprocess

Convert the dataset into the format required for the model:

cd data/preprocess
./process_dataset_for_model.sh <gpu_id>
cd ../../

This step generated AMR graphs using the SPRING model. Check their repository for more details.

Download the pretrained parameters of the adapters:

cd src
./download_pretrained_adapters.sh

Training

For training FactGraph using the FactCollect dataset, execute:

conda activate factgraph
./train.sh <gpu_id> 

Predicting

For predicting, run:

./predict.sh <checkpoint_folder> <gpu_id>

Training FactGraph - Edge-level

Preprocess

Download the files train.tsv and test.tsv from this link provided by Goyal and Durrett (2021). Copy those files to data\edge_level_data

Convert the dataset into the format required for the model:

cd data/preprocess
./process_dataset_for_edge_model.sh <gpu_id>
cd ../../

Training

For training FactGraph using the FactCollect dataset, execute:

conda activate factgraph
./train_edgelevel.sh <gpu_id>

Predicting

For predicting, run:

./predict_edgelevel.sh <checkpoint_folder> <gpu_id>

Trained Models

A FactGraph checkpoint trained on FactCollect dataset can be found here. Test set results:

 {'accuracy': 0.89, 'bacc': 0.8904, 'f1': 0.89, 'size': 600, 'cnndm': {'bacc': 0.7717, 'f1': 0.8649, 'size': 370}, 'xsum': {'bacc': 0.6833, 'f1': 0.9304, 'size': 230}}

A FactGraph-edge checkpoint trained on the Maynez dataset can be found here. This checkpoint was selected using the test set. Test set results:

 {'accuracy': 0.8371, 'bacc': 0.8447, 'f1': 0.8371, 'f1_macro': 0.7362, 'accuracy_edge': 0.6948, 'bacc_edge': 0.6592, 'f1_edge': 0.6948}

Security

See CONTRIBUTING for more information.

License Summary

The documentation is made available under under the CC-BY-NC-4.0 License. See the LICENSE file.

Citation

@inproceedings{ribeiro-etal-2022-factgraph,
    title = "FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations",
    author = "Ribeiro, Leonardo F. R.  and
      Liu, Mengwen  and
      Gurevych, Iryna and
      Dreyer Markus and
      Bansal, Mohit",
      booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
      year={2022}
}