ACL Findings2025
Mitigate Position Bias in LLMs via Scaling a Single Hidden States Channel
Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang, Yongfeng Huang, Lili Qiu
摘要
Long-context language models (LCLMs) can process long context, but still exhibit position bias, also known as "lost in the middle", which indicates placing key information in the middle of the context will significantly affect performance. To mitigating this, we first explore the micro-level manifestations of position bias, concluding that attention weights are a microlevel expression of position bias. Then we identify that, in addition to position embeddings, positional information in hidden states also contributes to position bias, and it manifests itself in specific channels of hidden states, called positional hidden states. Based on these, we propose a method to mitigate position bias by scaling positional hidden states. Experiments on NaturalQuestions Multi-document QA, KV retrieval and LongBench, using various models including RoPE models, context windowextended models, and Alibi models, demonstrate the effectiveness and generalizability of our approach. Our method can improve performance by up to 15.2% in "lost in the middle" benchmark by modifying just one channel of hidden states. Our code is available at https://aka.ms/PositionalHidden.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 3 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding 引用了 Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding 引用了 FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
- Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation 引用了 Measuring Massive Multitask Language Understanding
- Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation 引用了 Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding
引用它的顶会 Paper3
- DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question AnsweringJiakai Li, Rongzheng Wang, Yizhuo Ma, Shuang Liang 等NeurIPS 2025 · 8 次引用
- Hallucination Detection from Structural Reasoning ModelJianbo Sun, Pengkun YangICML 2026
- Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context AdaptationZichong Li, Chen Liang, Liliang Ren, Tuo Zhao 等ICML 2026
它引用的 Paper19
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou 等ICLR 2021 · 7,905 次引用
- FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 次引用
- Train Short, Test Long: Attention with Linear Biases Enables Input Length ExtrapolationOfir Press, Noah A. Smith, Mike LewisICLR 2022 · 1,168 次引用
- Large Language Models Can Be Easily Distracted by Irrelevant ContextFreda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales 等ICML 2023 · 970 次引用
- Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional EncodingZhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao 等NeurIPS 2024 · 99 次引用
相关 Paper
- Distance between Relevant Information Pieces Causes Bias in Long-Context LLMsRunchu Tian, Yanghao Li, Yuepeng Fu, Siyang Deng 等ACL Findings 2025 · 2 次引用
- Found in the middle: Calibrating Positional Attention Bias Improves Long Context UtilizationCheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang 等ACL Findings 2024 · 31 次引用
- Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding ScalingZhenghua Wang, Yiran Ding, Changze Lv, Yixin Wu 等ACL Findings 2026
- Wavelet-based Positional Representation for Long ContextYui Oka, Taku Hasegawa, Kyosuke Nishida, Kuniko SaitoICLR 2025
- On the token distance modeling ability of higher RoPE attention dimensionXiangyu Hong, Che Jiang, Biqing Qi, Fandong Meng 等EMNLP Findings 2024 · 2 次引用
