EMNLP Findings2024
Can We Instruct LLMs to Compensate for Position Bias?
Meiru Zhang, Zaiqiao Meng, Nigel Collier
Abstract
Position bias in large language models (LLMs) leads to difficulty in accessing information retrieved from the retriever, thus downgrading the effectiveness of Retrieval-Augmented Generation (RAG) approaches in open-question answering. Recent studies reveal that this bias is related to disproportional attention across the context. In this work, we examine how to direct LLMs to allocate more attention towards a selected segment of the context through prompting, aiming to compensate for the shortage of attention. We find that language models do not have relative position awareness of the context but can be directed by promoting instruction with an exact document index. Our analysis contributes to a deeper understanding of position bias in LLMs and provides a pathway to mitigate this bias by instruction, thus benefiting LLMs in locating and utilizing relevant information from retrieved documents in RAG applications. The code and data in our study have been made publicly available. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers2
Builds on16
- Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
- Efficient Streaming Language Models with Attention SinksGuangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han et al.ICLR 2024 · 1,714 citations
- Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
- Augmenting Language Models with Long-Term MemoryWeizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu et al.NeurIPS 2023 · 256 citations
- Make Your LLM Fully Utilize the ContextShengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng et al.NeurIPS 2024 · 212 citations
Related papers
- Position Engineering: Boosting Large Language Models through Positional Information ManipulationZhiyuan He, Huiqiang Jiang, Zilong Wang, Yuqing Yang et al.EMNLP 2024 · 3 citations
- Found in the middle: Calibrating Positional Attention Bias Improves Long Context UtilizationCheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang et al.ACL Findings 2024 · 31 citations
- GraphInsight: Unlocking Insights in Large Language Models for Graph Structure UnderstandingYukun Cao, Shuo Han, Zengyi Gao, Zezhong Ding et al.ACL 2025
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented GenerationThomas Merth, Qichen Fu, Mohammad Rastegari, Mahyar NajibiICML 2024 · 14 citations
- PEAR: Position-Embedding-Agnostic Attention Re-weighting Enhances Retrieval-Augmented Generation with Zero Inference OverheadTao Tan, Yining Qian, Ang Lv, Hongzhan Lin et al.WWW 2025 · 4 citations
