EMNLP Findings2025
Characterizing Positional Bias in Large Language Models: A Multi-Model Evaluation of Prompt Order Effects
Patrick Schilcher, Dominik Karasin, Michael Schöpf, Haisam Saleh, Antonela Tommasel, Markus Schedl
Abstract
Large Language Models (LLMs) are widely used for a variety of tasks such as text generation, ranking, and decision-making. However, their outputs can be influenced by various forms of biases. One such bias is positional bias, where models prioritize items based on their position within a given prompt rather than their content or quality, impacting on how LLMs interpret and weigh information, potentially compromising fairness, reliability, and robustness. To assess positional bias, we prompt a range of LLMs to generate descriptions for a list of topics, systematically permuting their order and analyzing variations in the responses. Our analysis shows that ranking position affects structural features and coherence, with some LLMs also reordering or omitting topics. Nonetheless, the impact of positional bias varies across different LLMs and topics, indicating an interplay with other related biases.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on9
- LLM Evaluators Recognize and Favor Their Own GenerationsArjun Panickssery, Samuel R. Bowman, Shi FengNeurIPS 2024 · 865 citations
- Benchmarking Cognitive Biases in Large Language Models as EvaluatorsRyan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park et al.ACL Findings 2024 · 73 citations
- Look at the First Sentence: Position Bias in Question AnsweringMiyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim et al.EMNLP 2020 · 51 citations
- Split and Merge: Aligning Position Biases in LLM-based EvaluatorsZongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu et al.EMNLP 2024 · 14 citations
- Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language ModelsSheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, Hsin-Hsi ChenACL Findings 2024 · 9 citations
Related papers
- Serial Position Effects of Large Language ModelsXiaobo Guo, Soroush VosoughiACL Findings 2025 · 11 citations
- Where to show Demos in Your Prompt: A Positional Bias of In-Context LearningKwesi A. Cobbina, Tianyi ZhouEMNLP 2025
- Order-Independence Without Fine TuningReid McIlroy-Young, Katrina Brown, Conlan Olson, Linjun Zhang et al.NeurIPS 2024 · 10 citations
- Large Language Models are not Fair EvaluatorsPeiyi Wang, Lei Li, Liang Chen, Zefan Cai et al.ACL 2024
- Inertia in Moral and Value Judgments of Large Language ModelsBruce W. Lee, Yeongheon Lee, Hyunsoo ChoACL 2026 · 5 citations
