ACL Findings2025
Mitigate Position Bias in LLMs via Scaling a Single Hidden States Channel
Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang, Yongfeng Huang, Lili Qiu
Abstract
Long-context language models (LCLMs) can process long context, but still exhibit position bias, also known as "lost in the middle", which indicates placing key information in the middle of the context will significantly affect performance. To mitigating this, we first explore the micro-level manifestations of position bias, concluding that attention weights are a microlevel expression of position bias. Then we identify that, in addition to position embeddings, positional information in hidden states also contributes to position bias, and it manifests itself in specific channels of hidden states, called positional hidden states. Based on these, we propose a method to mitigate position bias by scaling positional hidden states. Experiments on NaturalQuestions Multi-document QA, KV retrieval and LongBench, using various models including RoPE models, context windowextended models, and Alibi models, demonstrate the effectiveness and generalizability of our approach. Our method can improve performance by up to 15.2% in "lost in the middle" benchmark by modifying just one channel of hidden states. Our code is available at https://aka.ms/PositionalHidden.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding cites Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding cites FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation cites Measuring Massive Multitask Language Understanding
- Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation cites Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding
Cited by top-tier papers3
- DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question AnsweringJiakai Li, Rongzheng Wang, Yizhuo Ma, Shuang Liang et al.NeurIPS 2025 · 8 citations
- Hallucination Detection from Structural Reasoning ModelJianbo Sun, Pengkun YangICML 2026
- Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context AdaptationZichong Li, Chen Liang, Liliang Ren, Tuo Zhao et al.ICML 2026
Builds on19
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
- Train Short, Test Long: Attention with Linear Biases Enables Input Length ExtrapolationOfir Press, Noah A. Smith, Mike LewisICLR 2022 · 1,168 citations
- Large Language Models Can Be Easily Distracted by Irrelevant ContextFreda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales et al.ICML 2023 · 970 citations
- Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional EncodingZhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao et al.NeurIPS 2024 · 99 citations
Related papers
- Distance between Relevant Information Pieces Causes Bias in Long-Context LLMsRunchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng et al.ACL Findings 2025 · 2 citations
- Found in the middle: Calibrating Positional Attention Bias Improves Long Context UtilizationCheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang et al.ACL Findings 2024 · 31 citations
- Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding ScalingZhenghua Wang, Yiran Ding, Changze Lv, Yixin Wu et al.ACL Findings 2026
- Wavelet-based Positional Representation for Long ContextYui Oka, Taku Hasegawa, Kyosuke Nishida, Kuniko SaitoICLR 2025
- On the token distance modeling ability of higher RoPE attention dimensionXiangyu Hong, Che Jiang, Biqing Qi, Fandong Meng et al.EMNLP Findings 2024 · 2 citations
