SummTriver is a tool for evaluating summaries automatically without using reference summaries, unlike ROUGE. SummTriver, utilizes multiple divergences to achieve this and has shown promising results, especially in data from summarization campaigns.
Now you can get SummTriver either through GitHub or Docker. You can know more about them in our new and growing Wiki.
Check our Wiki where you'll find out more information about SummTriver.
If you use this program, please cite it with:
L. A. Cabrera-Diego and J.-M. Torres-Moreno, "SummTriver: A new trivergent model to evaluate summaries automatically without human references", Data & Knowledge Engineering, vol. 113, pp. 184–197, 2018. https://doi.org/10.1016/j.datak.2017.09.001
@article{CABRERADIEGO2018184,
title = "SummTriver: A new trivergent model to evaluate summaries automatically without human references",
journal = "Data & Knowledge Engineering",
volume = "113",
pages = "184 - 197",
year = "2018",
issn = "0169-023X",
doi = "doi.org/10.1016/j.datak.2017.09.001",
author = "Luis Adrián Cabrera-Diego and Juan-Manuel Torres-Moreno"
}
If you use UltraStemming for processing documents please cite as well:
J.-M. Torres-Moreno, "Beyond Stemming and Lemmatization: Ultra-stemming to Improve Automatic Text Summarization", CoRR, 2012. http://arxiv.org/abs/1209.3126
@article{DBLP:journals/corr/abs-1209-3126,
author = {Juan{-}Manuel Torres{-}Moreno},
title = {Beyond Stemming and Lemmatization: Ultra-stemming to Improve Automatic
Text Summarization},
journal = {CoRR},
volume = {abs/1209.3126},
year = {2012},
url = {http://arxiv.org/abs/1209.3126},
archivePrefix = {arXiv},
eprint = {1209.3126}
}
The examples were obtained from "PUCES" a French corpus of Automatic Text Summarization (http://juanmanuel.torres.free.fr/corpus/puces/):
L. A. Cabrera-Diego, J.-M. Torres-Moreno, and B. Durette, "Evaluating Multiple Summaries Without Human Models: A First Experiment with a Trivergent Model" in Natural Language Processing and Information Systems. Proceedings of the 21st International conference on the Application of Natural Language to Information Systems (NLDB 2016), Manchester, UK, 2016, vol. 9612, pp. 91–101. https://doi.org/10.1007/978-3-319-41754-7_8
@inproceedings{cabrera-diego_evaluating_2016,
address = {Manchester, UK},
title = {Evaluating {Multiple} {Summaries} {Without} {Human} {Models}: {A} {First} {Experiment} with a {Trivergent} {Model}},
volume = {9612},
booktitle = {Natural {Language} {Processing} and {Information} {Systems}. {Proceedings} of the 21st {International} conference on the {Application} of {Natural} {Language} to {Information} {Systems} ({NLDB} 2016)},
author = {Cabrera-Diego, Luis Adrián and Torres-Moreno, Juan-Manuel and Durette, Barthélémy},
editor = {Métais, Elisabeth and Meziane, Farid and Saraee, Mohamed and Sugumaran, Vijay and Vadera, Sunil},
year = {2016},
doi = "doi.org/10.1007/978-3-319-41754-7_8"
pages = {91--101}
}