Lune

NeurIPS2021Top-tier venue

Global-aware Beam Search for Neural Abstractive Summarization

Ye Ma, Zixun Lan, Lu Zong, Kaizhu Huang

Abstract

This study develops a calibrated beam-based algorithm with awareness of the global attention distribution for neural abstractive summarization, aiming to improve the local optimality problem of the original beam search in a rigorous way. Specifically, a novel global protocol is proposed based on the attention distribution to stipulate how a global optimal hypothesis should attend to the source. A global scoring mechanism is then developed to regulate beam search to generate summaries in a near-global optimal fashion. This novel design enjoys a distinctive property, i.e., the global attention distribution could be predicted before inference, enabling step-wise improvements on the beam search through the global scoring mechanism. Extensive experiments on nine datasets show that the global (attention)-aware inference significantly improves state-of-the-art summarization models even using empirical hyper-parameters. The algorithm is also proven robust as it remains to generate meaningful texts with corrupted attention distributions. The codes and a comprehensive set of examples are available. 2

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human EvaluationYixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao et al.ACL 2023 · 50 citations

Builds on6

  1. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  2. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive SummarizationJingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. LiuICML 2020 · 2,453 citations
  3. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  4. AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed GradientsJuntang Zhuang, Tommy Tang, Yifan Ding, Sekhar Tatikonda et al.NeurIPS 2020 · 697 citations
  5. VMSMO: Learning to Generate Multimodal Summary for Video-based News ArticlesMingzhe Li, Xiuying Chen, Shen Gao, Zhangming Chan et al.EMNLP 2020 · 65 citations

Related papers

  1. Attend, Translate and Summarize: An Efficient Method for Neural Cross-Lingual SummarizationJunnan Zhu, Yu Zhou, Jiajun Zhang, Chengqing ZongACL 2020 · 51 citations
  2. Enriching and Controlling Global Semantics for Text SummarizationThong Nguyen, Anh Tuan Luu, Truc Lu, Tho QuanEMNLP 2021 · 26 citations
  3. BRIO: Bringing Order to Abstractive SummarizationYixin Liu, Pengfei Liu, Dragomir R. Radev, Graham NeubigACL 2022 · 329 citations
  4. Towards Summary Candidates FusionMathieu Ravaut, Shafiq R. Joty, Nancy F. ChenEMNLP 2022 · 10 citations
  5. Enhancing Abstractiveness of Summarization Models through Calibrated DistillationHwanjun Song, Igor Shalyminov, Hang Su, Siffi Singh et al.EMNLP Findings 2023 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines