Lune

EMNLP Findings2021

MiRANews: Dataset and Benchmarks for Multi-Resource-Assisted News Summarization

Xinnuo Xu, Ondrej Dusek, Shashi Narayan, Verena Rieser, Ioannis Konstas

Abstract

One of the most challenging aspects of current single-document news summarization is that the summary often contains 'extrinsic hallucinations', i.e., facts that are not present in the source document, which are often derived via world knowledge.This causes summarization systems to act more like open-ended language models tending to hallucinate facts that are erroneous.In this paper, we mitigate this problem with the help of multiple supplementary resource documents assisting the task.We present a new dataset MIRANEWS and benchmark existing summarization models. 1 In contrast to multi-document summarization, which addresses multiple events from several source documents, we still aim at generating a summary for a single document.We show via data analysis that it's not only the models which are to blame: more than 27% of facts mentioned in the gold summaries of MIRANEWS are better grounded on assisting documents than in the main source articles.An error analysis of generated summaries from pretrained models fine-tuned on MIRANEWS reveals that this has an even bigger effects on models: assisted summarization reduces 55% of hallucinations when compared to single-document summarization models trained on the main article only.Summary: Kathy Griffin and Howard Stern gather to say goodbye at Joan Rivers funeral in manhattan New York AP.Even in death, Joan Rivers got what she wanted: a star-studded funeral, with the worlds of Hollywood, fashion, media and money all among the mourners on Sunday morning.Document: on sunday morning, a legion of notables turned out at new york's temple emanu-el to remember rivers, who died thursday at 81: kathy griffin, whose edgy, biting comedy career was largely made possible by rivers; colleague and friend kelly osbourne; sarah jessica parker and whoopi goldberg; howard stern... lined up outside the fifth avenue synagogue and waited for their names to be checked against a list before entering.barricades lined several blocks of manhattan's fifth avenue, and a crowd of fans and media stood watch across the street.the comedian detailed in her 2012 book i hate everyone starting with me'' that she hoped fora huge showbiz affair with lights, cameras, action ''and ``hollywood all the way.''... Assisting Document: new york city's temple emanu-el; joan rivers all access photo/splash news online updated 09

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports cites BERTScore: Evaluating Text Generation with BERT
  • On Extractive and Abstractive Neural Document Summarization with Transformer Language Models cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
  • PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization cites On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on12

  1. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  2. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive SummarizationJingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. LiuICML 2020 · 2,453 citations
  3. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  4. On Extractive and Abstractive Neural Document Summarization with Transformer Language ModelsJonathan Pilault, Raymond Li, Sandeep Subramanian, Chris PalEMNLP 2020 · 186 citations
  5. Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology ReportsYuhao Zhang, Derek Merck, Emily Bao Tsai, Christopher D. Manning et al.ACL 2020 · 160 citations

Related papers

  1. GlobeSumm: A Challenging Benchmark Towards Unifying Multi-lingual, Cross-lingual and Multi-document News SummarizationYangfan Ye, Xiachong Feng, Xiaocheng Feng, Weitao Ma et al.EMNLP 2024 · 8 citations
  2. SumREN: Summarizing Reported Speech about Events in NewsRevanth Gangi Reddy, Heba Elfardy, Hou Pong Chan, Kevin Small et al.AAAI 2023 · 7 citations
  3. ASPECTNEWS: Aspect-Oriented Summarization of News DocumentsOjas Ahuja, Jiacheng Xu, Akshay Gupta, Kevin Horecka et al.ACL 2022
  4. Faithful to the Document or to the World? Mitigating Hallucinations via Entity-Linked Knowledge in Abstractive SummarizationYue Dong, John Wieting, Pat VergaEMNLP Findings 2022 · 16 citations
  5. Multilingual Fine-Grained News Headline Hallucination DetectionJiaming Shen, Tianqi Liu, Jialu Liu, Zhen Qin et al.EMNLP Findings 2024 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines