Lune

EMNLP Findings2025

Characterizing Positional Bias in Large Language Models: A Multi-Model Evaluation of Prompt Order Effects

Patrick Schilcher, Dominik Karasin, Michael Schöpf, Haisam Saleh, Antonela Tommasel, Markus Schedl

Abstract

Large Language Models (LLMs) are widely used for a variety of tasks such as text generation, ranking, and decision-making. However, their outputs can be influenced by various forms of biases. One such bias is positional bias, where models prioritize items based on their position within a given prompt rather than their content or quality, impacting on how LLMs interpret and weigh information, potentially compromising fairness, reliability, and robustness. To assess positional bias, we prompt a range of LLMs to generate descriptions for a list of topics, systematically permuting their order and analyzing variations in the responses. Our analysis shows that ranking position affects structural features and coherence, with some LLMs also reordering or omitting topics. Nonetheless, the impact of positional bias varies across different LLMs and topics, indicating an interplay with other related biases.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LLM Evaluators Recognize and Favor Their Own Generations cites Benchmarking Cognitive Biases in Large Language Models as Evaluators
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on9

  1. LLM Evaluators Recognize and Favor Their Own GenerationsArjun Panickssery, Samuel R. Bowman, Shi FengNeurIPS 2024 · 865 citations
  2. Benchmarking Cognitive Biases in Large Language Models as EvaluatorsRyan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park et al.ACL Findings 2024 · 73 citations
  3. Look at the First Sentence: Position Bias in Question AnsweringMiyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim et al.EMNLP 2020 · 51 citations
  4. Split and Merge: Aligning Position Biases in LLM-based EvaluatorsZongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu et al.EMNLP 2024 · 14 citations
  5. Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language ModelsSheng-Lun Wei, Cheng-Kuang Wu, Hen-Hsen Huang, Hsin-Hsi ChenACL Findings 2024 · 9 citations

Related papers

  1. Serial Position Effects of Large Language ModelsXiaobo Guo, Soroush VosoughiACL Findings 2025 · 11 citations
  2. Where to show Demos in Your Prompt: A Positional Bias of In-Context LearningKwesi A. Cobbina, Tianyi ZhouEMNLP 2025
  3. Order-Independence Without Fine TuningReid McIlroy-Young, Katrina Brown, Conlan Olson, Linjun Zhang et al.NeurIPS 2024 · 10 citations
  4. Large Language Models are not Fair EvaluatorsPeiyi Wang, Lei Li, Liang Chen, Zefan Cai et al.ACL 2024
  5. Inertia in Moral and Value Judgments of Large Language ModelsBruce W. Lee, Yeongheon Lee, Hyunsoo ChoACL 2026 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines