Global-aware Beam Search for Neural Abstractive Summarization
Abstract
This study develops a calibrated beam-based algorithm with awareness of the global attention distribution for neural abstractive summarization, aiming to improve the local optimality problem of the original beam search in a rigorous way. Specifically, a novel global protocol is proposed based on the attention distribution to stipulate how a global optimal hypothesis should attend to the source. A global scoring mechanism is then developed to regulate beam search to generate summaries in a near-global optimal fashion. This novel design enjoys a distinctive property, i.e., the global attention distribution could be predicted before inference, enabling step-wise improvements on the beam search through the global scoring mechanism. Extensive experiments on nine datasets show that the global (attention)-aware inference significantly improves state-of-the-art summarization models even using empirical hyper-parameters. The algorithm is also proven robust as it remains to generate meaningful texts with corrupted attention distributions. The codes and a comprehensive set of examples are available. 2
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Cited by top-tier papers1
Builds on6
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive SummarizationJingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. LiuICML 2020 · 2,453 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed GradientsJuntang Zhuang, Tommy Tang, Yifan Ding, Sekhar Tatikonda et al.NeurIPS 2020 · 697 citations
- VMSMO: Learning to Generate Multimodal Summary for Video-based News ArticlesMingzhe Li, Xiuying Chen, Shen Gao, Zhangming Chan et al.EMNLP 2020 · 65 citations
Related papers
- Attend, Translate and Summarize: An Efficient Method for Neural Cross-Lingual SummarizationJunnan Zhu, Yu Zhou, Jiajun Zhang, Chengqing ZongACL 2020 · 51 citations
- Enriching and Controlling Global Semantics for Text SummarizationThong Nguyen, Anh Tuan Luu, Truc Lu, Tho QuanEMNLP 2021 · 26 citations
- BRIO: Bringing Order to Abstractive SummarizationYixin Liu, Pengfei Liu, Dragomir R. Radev, Graham NeubigACL 2022 · 329 citations
- Towards Summary Candidates FusionMathieu Ravaut, Shafiq R. Joty, Nancy F. ChenEMNLP 2022 · 10 citations
- Enhancing Abstractiveness of Summarization Models through Calibrated DistillationHwanjun Song, Igor Shalyminov, Hang Su, Siffi Singh et al.EMNLP Findings 2023 · 3 citations
