ACL Findings2026
Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalization
Mingkuan Zhao, Wentao Hu, Tianchen Huang, Yuheng Min, Suquan Chen, Yide Gao, Yanbo Zhai, Shuangyong Song, Xuelong Li
Abstract
Hallucination in Large Language Models (LLMs)-characterized by the generation of content inconsistent with contextual facts or logical constraints-remains a persistent challenge for reliable deployment. In this work, we address this issue through a geometric framework rooted in the linear representation hypothesis. We propose that hallucinations manifest as orthogonal noise relative to the semantic manifold of the residual stream. Specifically, we hypothesize that while attention heads ideally propagate information congruent with the context subspace, hallucinations arise when specific heads introduce components orthogonal to this subspace, disrupting the coherence of the latent representation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- The Linear Representation Hypothesis and the Geometry of Large Language Models cites Locating and Editing Factual Associations in GPT
- Inference-Time Intervention: Eliciting Truthful Answers from a Language Model cites Locating and Editing Factual Associations in GPT
- Inference-Time Intervention: Eliciting Truthful Answers from a Language Model cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
Builds on21
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- Inference-Time Intervention: Eliciting Truthful Answers from a Language ModelKenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister et al.NeurIPS 2023 · 1,549 citations
- The Linear Representation Hypothesis and the Geometry of Large Language ModelsKiho Park, Yo Joong Choe, Victor VeitchICML 2024 · 461 citations
Related papers
- Whispers that Shake Foundations: Analyzing and Mitigating False Premise Hallucinations in Large Language ModelsHongbang Yuan, Pengfei Cao, Zhuoran Jin, Yubo Chen et al.EMNLP 2024 · 5 citations
- One SPACE to Rule Them All: Jointly Mitigating Factuality and Faithfulness Hallucinations in LLMsPengbo Wang, Chaozhuo Li, Chenxu Wang, Liwen Zheng et al.NeurIPS 2025
- NDAD: Negative-Direction Aware Decoding for Large Language Models via Controllable Hallucination Signal InjectionPanjia Qiu, Mingyuan Fan, Cen Chen, Daixin WangICLR 2026
- Why LLMs Hallucinate on Structured Knowledge: A Mechanistic Analysis of Reasoning over Linearized RepresentationsShanghao Li, Jinda Han, Yibo Wang, Yuanjie Zhu et al.ACL 2026
- The Digital Dunning-Kruger Effect: Decoupling Hallucinations via Geometric Hidden-state Observation for Semantic TruthfulnessYueheng Mao, Min Yu, Gengwang Li, Jianguo Jiang et al.ACL 2026
