ACL Findings2023
Alleviating Exposure Bias via Multi-level Contrastive Learning and Deviation Simulation in Abstractive Summarization
Jiawen Xie, Qi Su, Shaoting Zhang, Xiaofan Zhang
Abstract
Most Transformer based abstractive summarization systems have a severe mismatch between training and inference, i.e., exposure bias.From diverse perspectives, we introduce a simple multi-level contrastive learning framework for abstractive summarization (SimMCS) and a tailored sparse decoder self-attention pattern (SDSA) to bridge the gap between training and inference to improve model performance.Compared with previous contrastive objectives focusing only on the relative order of probability mass assigned to non-gold summaries, SimMCS additionally takes their absolute positions into account, which guarantees that the relatively high-quality (positive) summaries among them can be properly assigned high probability mass, and further enhances the capability of discriminating summary quality beyond exploiting potential artifacts of specific metrics.SDSA simulates the possible inference scenarios of deviation in the training phase to get closer to the ideal paradigm.Our approaches outperform the previous state-ofthe-art results on two summarization datasets while just adding fairly low overhead.Further empirical analysis shows our model preserves the advantages of prior contrastive methods and possesses strong few-shot learning ability.Our code is available at https://github.com/xjw- nlp/SimMCS.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- BARTScore: Evaluating Generated Text as Text Generation cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- BARTScore: Evaluating Generated Text as Text Generation cites BERTScore: Evaluating Text Generation with BERT
- BARTScore: Evaluating Generated Text as Text Generation cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
- On Context Utilization in Summarization with Large Language Models cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- On Context Utilization in Summarization with Large Language Models cites BERTScore: Evaluating Text Generation with BERT
Cited by top-tier papers1
Builds on22
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive SummarizationJingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. LiuICML 2020 · 2,453 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- BARTScore: Evaluating Generated Text as Text GenerationWeizhe Yuan, Graham Neubig, Pengfei LiuNeurIPS 2021 · 1,143 citations
Related papers
- Sequence Level Contrastive Learning for Text SummarizationShusheng Xu, Xingxing Zhang, Yi Wu, Furu WeiAAAI 2022 · 113 citations
- Give the Truth: Incorporate Semantic Slot into Abstractive Dialogue SummarizationLulu Zhao, Weihao Zeng, Weiran Xu, Jun GuoEMNLP Findings 2021 · 10 citations
- CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive SummarizationShuyang Cao, Lu WangEMNLP 2021 · 130 citations
- Cross-Lingual Abstractive Summarization with Limited Parallel ResourcesYu Bai, Yang Gao, Heyan HuangACL 2021
- Discriminative Adversarial Search for Abstractive SummarizationThomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski et al.ICML 2020 · 37 citations
