Lune

ACL Findings2024

Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?

Marcio Fonseca, Shay B. Cohen

Abstract

In this work, we investigate the controllability of large language models (LLMs) on scientific summarization tasks. We identify key stylistic and content coverage factors that characterize different types of summaries such as paper reviews, abstracts, and lay summaries. By controlling stylistic features, we find that non-finetuned LLMs outperform humans in the MuP review generation task, both in terms of similarity to reference summaries and human preferences. Also, we show that we can improve the controllability of LLMs with keyword-based classifier-free guidance (CFG) while achieving lexical overlap comparable to strong fine-tuned baselines on arXiv and PubMed. However, our results also indicate that LLMs cannot consistently generate long summaries with more than 8 sentences. Furthermore, these models exhibit limited capacity to produce highly abstractive lay summaries. Although LLMs demonstrate strong generic summarization competency, sophisticated content control without costly finetuning remains an open problem for domainspecific applications. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Training language models to follow instructions with human feedback cites Multitask Prompted Training Enables Zero-Shot Task Generalization
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on16

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  3. Big Bird: Transformers for Longer SequencesManzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie et al.NeurIPS 2020 · 3,159 citations
  4. Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations
  5. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations

Related papers

  1. Understanding LLMs' summarization capabilities: an analysis of biomedical abstract and lay summary generationBatuhan Nursal, Cassie S. MitchellACL Findings 2026
  2. Model-based Preference Optimization in Abstractive Summarization without Human FeedbackJaepill Choi, Kyubyung Chae, Jiwoo Song, Yohan Jo et al.EMNLP 2024 · 1 citation
  3. Control Large Language Models via Divide and ConquerBingxuan Li, Yiwei Wang, Tao Meng, Kai-Wei Chang et al.EMNLP 2024 · 1 citation
  4. Prompt-Based Length Controlled Generation with Multiple Control TypesRenlong Jie, Xiaojun Meng, Lifeng Shang, Xin Jiang et al.ACL Findings 2024 · 4 citations
  5. Large Language Models are Not Yet Human-Level Evaluators for Abstractive SummarizationChenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You et al.EMNLP Findings 2023 · 52 citations
Dusk over the sea between two cliffs drawn in fine vertical lines