Prompt Injection as Role Confusion
Charles Ye, Jasmine Cui, Dylan Hadfield-Menell
Abstract
LLMs see the world as a single stream of text, partitioned into roles like <user> or <tool>. We trace prompt injection to role confusion: models perceive the source of text from how it sounds, not its labeled role. A command hidden in a webpage hijacks an agent simply because it sounds like <user> text, despite its <tool> label. We design role probes to measure how LLMs internally perceive "who is speaking," and find that injected text occupies the same representational space as the trusted role it imitates. We demonstrate this with CoT Forgery, a zero-shot attack that injects fabricated reasoning into user prompts and tool outputs. Models mistake the forgery for their own thoughts, yielding 60% attack success against frontier models with near-zero baselines. Strikingly, the degree of role confusion predicts attack success before a single token is generated. This mechanism generalizes beyond CoT Forgery to standard agent prompt injections, revealing prompt injection as a measurable consequence of role perception. To the model, sounding like a role is indistinguishable from being one. Project page at role-confusion.github.io.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext dd6a6fd6-957e-44d0-a98d-9deebe19a05fBuilds on18
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought PromptingMiles Turpin, Julian Michael, Ethan Perez, Samuel R. BowmanNeurIPS 2023 · 1,792 citations
- Many-shot JailbreakingCem Anil, Esin Durmus, Nina Panickssery, Mrinank Sharma et al.NeurIPS 2024 · 338 citations
- The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt InjectionsMilad Nasr, Nicholas Carlini, Chawin Sitawarin, Sander V. Schulhoff et al.USENIX Security 2026 · 134 citations
- Tensor Trust: Interpretable Prompt Injection Attacks from an Online GameSam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato et al.ICLR 2024 · 123 citations
Related papers
- The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)Zihao Wang, Yibo Jiang, Jiahao Yu, Heqing HuangICML 2025
- ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source DataReachal Wang, Yuqi Jia, Neil Zhenqiang GongNDSS 2026 · 24 citations
- ChatInject: Abusing Chat Templates for Prompt Injection in LLM AgentsHwan Chang, Yonghyun Jun, Hwanhee LeeICLR 2026 · 32 citations
- Context Contamination in LLM Analysis of Network Security Logs: Poison with Passive Prompt Injection and Mitigation EvaluationRabimba Karanjai, Yang Lu, Hemanth Hegadehalli Madhavarao, Lei Xu et al.USENIX Security 2026 · 4 citations
- PromptLocate: Localizing Prompt Injection AttacksYuqi Jia, Yupei Liu, Zedian Shao, Jinyuan Jia et al.S&P 2026 · 35 citations
