Lune

ACL2022Top-tier venue

PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization

Wen Xiao, Iz Beltagy, Giuseppe Carenini, Arman Cohan

Abstract

We introduce PRIMERA, a pre-trained model for multi-document representation with a focus on summarization that reduces the need for dataset-specific architectures and large amounts of fine-tuning labeled data. PRIMERA uses our newly proposed pre-training objective designed to teach the model to connect and aggregate information across documents. It also uses efficient encoder-decoder transformers to simplify the processing of concatenated input documents. With extensive experiments on 6 multi-document summarization datasets from 3 different domains on zero-shot, few-shot and full-supervised settings, PRIMERA outperforms current state-of-the-art dataset-specific and pre-trained models on most of these settings with large margins. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

22 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Towards Unifying Multi-Lingual and Cross-Lingual Summarization cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Towards Unifying Multi-Lingual and Cross-Lingual Summarization cites Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization
  • Towards Unifying Multi-Lingual and Cross-Lingual Summarization cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
  • Big Bird: Transformers for Longer Sequences cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
  • Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
  • Repository-Level Prompt Generation for Large Language Models of Code cites FLEX: Unifying Evaluation for Few-Shot NLP
  • Leveraging Locality in Abstractive Text Summarization cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Leveraging Locality in Abstractive Text Summarization cites Big Bird: Transformers for Longer Sequences
  • Leveraging Locality in Abstractive Text Summarization cites PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
  • Unlimiformer: Long-Range Transformers with Unlimited Length Input cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  • Unlimiformer: Long-Range Transformers with Unlimited Length Input cites Big Bird: Transformers for Longer Sequences
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers22

  1. Repository-Level Prompt Generation for Large Language Models of CodeDisha Shrivastava, Hugo Larochelle, Daniel TarlowICML 2023 · 184 citations
  2. Unlimiformer: Long-Range Transformers with Unlimited Length InputAmanda Bertsch, Uri Alon, Graham Neubig, Matthew GormleyNeurIPS 2023 · 176 citations
  3. Towards Unifying Multi-Lingual and Cross-Lingual SummarizationJiaan Wang, Fandong Meng, Duo Zheng, Yunlong Liang et al.ACL 2023 · 24 citations
  4. Leveraging Locality in Abstractive Text SummarizationYixin Liu, Ansong Ni, Linyong Nan, Budhaditya Deb et al.EMNLP 2022 · 19 citations
  5. Summary-Oriented Vision Modeling for Multimodal Abstractive SummarizationYunlong Liang, Fandong Meng, Jinan Xu, Jiaan Wang et al.ACL 2023 · 17 citations

Builds on13

  1. Big Bird: Transformers for Longer SequencesManzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie et al.NeurIPS 2020 · 3,159 citations
  2. PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive SummarizationJingqing Zhang, Yao Zhao, Mohammad Saleh, Peter J. LiuICML 2020 · 2,453 citations
  3. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  4. Leveraging Graph to Improve Abstractive Multi-Document SummarizationWei Li, Xinyan Xiao, Jiachen Liu, Hua Wu et al.ACL 2020 · 118 citations
  5. FLEX: Unifying Evaluation for Few-Shot NLPJonathan Bragg, Arman Cohan, Kyle Lo, Iz BeltagyNeurIPS 2021 · 114 citations

Related papers

  1. A Hierarchical Encoding-Decoding Scheme for Abstractive Multi-document SummarizationChenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You et al.EMNLP Findings 2023 · 18 citations
  2. Peek Across: Improving Multi-Document Modeling via Cross-Document Question-AnsweringAvi Caciularu, Matthew E. Peters, Jacob Goldberger, Ido Dagan et al.ACL 2023 · 8 citations
  3. CDLM: Cross-Document Language ModelingAvi Caciularu, Arman Cohan, Iz Beltagy, Matthew E. Peters et al.EMNLP Findings 2021 · 56 citations
  4. Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue SummarizationWeixiao Zhou, Gengyao Li, Xianfu Cheng, Xinnian Liang et al.EMNLP Findings 2023 · 1 citation
  5. Pre-training via ParaphrasingMike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan et al.NeurIPS 2020 · 165 citations
Dusk over the sea between two cliffs drawn in fine vertical lines