Lune

ACL Findings2025

Behavioral Analysis of Information Salience in Large Language Models

Jan Trienes, Jörg Schlötterer, Junyi Jessy Li, Christin Seifert

Abstract

Large Language Models (LLMs) excel at text summarization, a task that requires models to select content based on its importance. However, the exact notion of salience that LLMs have internalized remains unclear. To bridge this gap, we introduce an explainable framework to systematically derive and investigate information salience in LLMs through their summarization behavior. Using length-controlled summarization as a behavioral probe into the content selection process, and tracing the answerability of Questions Under Discussion throughout, we derive a proxy for how models prioritize information. Our experiments on 13 models across four datasets reveal that LLMs have a nuanced, hierarchical notion of salience, generally consistent across model families and sizes. While models show highly consistent behavior and hence salience patterns, this notion of salience cannot be accessed through introspection, and only weakly correlates with human perceptions of information salience.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Related papers

  1. SelfIE: Self-Interpretation of Large Language Model EmbeddingsHaozhe Chen, Carl Vondrick, Chengzhi MaoICML 2024 · 58 citations
  2. Towards a Holistic and Automated Evaluation Framework for Multi-Level Comprehension of LLMs in Book-Length ContextsYuho Lee, Jiaqi Deng, Nicole Hee-Yeon Kim, Hyangsuk Min et al.EMNLP 2025
  3. On Context Utilization in Summarization with Large Language ModelsMathieu Ravaut, Aixin Sun, Nancy F. Chen, Shafiq JotyACL 2024
  4. On Behavioral Alignment of Model-Code and Human-Code Understandability via Behavioral ProxiesXiaokai Rong, Aashish Yadavally, Hridya Dhulipala, Anh H. N. Nguyen et al.ISSTA 2026
  5. Length Representations in Large Language ModelsSangjun Moon, Dasom Choi, Jingun Kwon, Hidetaka Kamigaito et al.EMNLP Findings 2025
Dusk over the sea between two cliffs drawn in fine vertical lines