Lune

ACL Findings2026

Lost in the Prompt Order: Revealing the Limitations of Causal Attention in Language Models

Hyunjong Ok, Jaeho Lee

摘要

Large language models exhibit surprising sensitivity to the structure of the prompt, but the mechanisms underlying this sensitivity remain poorly understood. In this work, we conduct an in-depth investigation on a striking case: in multiple-choice question answering, placing context before the questions and options (CQO) outperforms the reverse order (QOC) by over 14%p, consistently over a wide range of models and datasets. Through systematic architectural analysis, we identify causal attention as the core mechanism: in QOC prompts, the causal mask prevents option tokens from attending to context, creating an information bottleneck where context becomes invisible to options. Some Cantonese don't like chili, so some southerners don't like chili. A.Some Cantonese love chili B.Some people who like peppers are (...) C.All Cantonese are southerners D.Some Cantonese like neither peppers (...) Which of the following can guarantee the above argument? Some Cantonese don't like chili, so some southerners don't like chili. A.Some Cantonese love chili B.Some people who like peppers are (...) C.All Cantonese are southerners D.Some Cantonese like neither peppers (...) Which of the following can guarantee the above argument? Context Question Option QOC 14.7 % Drop!

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

可以从这些问题问起

Lune 可以查阅

全文 · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

它引用的引用它的从左到右按年份排列,引用越多,位置越高。

它引用的 Paper1

  1. Make Your LLM Fully Utilize the ContextShengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng 等NeurIPS 2024 · 212 次引用

相关 Paper

  1. COPO: Causal-Oriented Policy Optimization for Hallucinations of MLLMsPeizheng Guo, Jingyao Wang, Wenwen Qiang, Jiahuan Zhou 等CVPR 2026 · 1 次引用
  2. Rapid Selection and Ordering of In-Context Demonstrations via Prompt Embedding ClusteringKha Pham, Hung Le, Man Ngo, Truyen TranICLR 2025
  3. Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?Nishant Balepur, Abhilasha Ravichander, Rachel RudingerACL 2024 · 5 次引用
  4. Segment-Based Attention Masking for GPTsShahar Katz, Liran Ringel, Yaniv Romano, Lior WolfACL 2025
  5. Leveraging Large Language Models for Multiple Choice Question AnsweringJoshua Robinson, David WingateICLR 2023 · 40 次引用
黄昏的海面,两侧是细线勾勒的悬崖