EMNLP Findings2025
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, Aram Galstyan, Azton Wells, Roy Schwartz, Eliu A. Huerta, Hao Peng
Abstract
Large language models (LLMs) often fail to scale their performance on long-context tasks performance in line with the context lengths they support. This gap is commonly attributed to retrieval failures-the models' inability to identify relevant information in the long inputs. Accordingly, recent efforts often focus on evaluating and improving LLMs' retrieval performance: if retrieval is perfect, a model should, in principle, perform just as well on a long input as it does on a short one-or should it? This paper presents findings that the answer to this question may be negative. Our systematic experiments across 5 open-and closedsource LLMs on math, question answering, and coding tasks reveal that, even when models can perfectly retrieve all relevant information, their performance still degrades substantially (13.9%-85%) as input length increases but remains well within the models' claimed lengths. This failure occurs even when the irrelevant tokens are replaced with minimally distracting whitespace, and, more surprisingly, when they are all masked and the models are forced to attend only to the relevant tokens. A similar performance drop is observed when all relevant evidence is placed immediately before the question. Our findings reveal a previouslyunrealized limitation: the sheer length of the input alone can hurt LLM performance, independent of retrieval quality and without any distraction. They motivate our simple, modelagnostic mitigation strategy that transforms a long-context task into a short-context one by prompting the model to recite the retrieved evidence before attempting to solve the problem. On RULER, we observe a consistent improvement of GPT-4o up to 4% on an already strong baseline.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
10 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- The First Drop of Ink: Nonlinear Impact of Misleading Information in Long-Context Reasoning cites Large Language Models Can Be Easily Distracted by Irrelevant Context
- The First Drop of Ink: Nonlinear Impact of Misleading Information in Long-Context Reasoning cites The Power of Noise: Redefining Retrieval for RAG Systems
- YaRN: Efficient Context Window Extension of Large Language Models cites Measuring Massive Multitask Language Understanding
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models cites YaRN: Efficient Context Window Extension of Large Language Models
Cited by top-tier papers10
- Structurally Aligned Subtask-Level Memory for Software Engineering AgentsKangning Shen, Jingyuan Zhang, Chenxi Sun, Wencong Zeng et al.ICML 2026 · 9 citations
- GAM: Hierarchical Graph-based Agentic Memory for LLM AgentsZhaofen Wu, Hanrong Zhang, Fulin Lin, Wujiang Xu et al.ACL 2026 · 8 citations
- The First Drop of Ink: Nonlinear Impact of Misleading Information in Long-Context ReasoningMuhan Gao, Zih-Ching Chen, Kuan-Hao HuangICML 2026 · 1 citation
- APEX-MEM: Agentic Semi-Structured Memory with Temporal Reasoning for Long-Term Conversational AIPratyay Banerjee, Masud Moshtaghi, Shivashankar Subramanian, Amita Misra et al.ACL 2026 · 1 citation
- Can Structural Cues Save LLMs? Evaluating Language Models in Massive Document StreamsYukyung Lee, Yebin Lim, Woojun Jung, Wonjun Choi et al.KDD 2026
Builds on26
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Large Language Models Can Be Easily Distracted by Irrelevant ContextFreda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales et al.ICML 2023 · 970 citations
- YaRN: Efficient Context Window Extension of Large Language ModelsBowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico ShippoleICLR 2024 · 508 citations
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language ModelsYukang Chen, Shengju Qian, Haotian Tang, Xin Lai et al.ICLR 2024 · 254 citations
- The Power of Noise: Redefining Retrieval for RAG SystemsFlorin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice et al.SIGIR 2024 · 212 citations
Related papers
- Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language ModelsMosh Levy, Alon Jacoby, Yoav GoldbergACL 2024 · 77 citations
- Ruler: A Model-Agnostic Method to Control Generated Length for Large Language ModelsJiaming Li, Lei Zhang, Yunshui Li, Ziqiang Liu et al.EMNLP Findings 2024 · 1 citation
- NoLiMa: Long-Context Evaluation Beyond Literal MatchingAli Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui et al.ICML 2025
- Needle Threading: Can LLMs Follow Threads Through Near-Million-Scale Haystacks?Jonathan Roberts, Kai Han, Samuel AlbanieICLR 2025
- Can't Remember Details in Long Documents? You Need Some R&RDevanshu Agrawal, Shang Gao, Martin GajekEMNLP Findings 2024 · 2 citations
