ACL Findings2024
Understanding and Patching Compositional Reasoning in LLMs
Zhaoyi Li, Gangwei Jiang, Hong Xie, Linqi Song, Defu Lian, Ying Wei
Abstract
LLMs have marked a revolutonary shift, yet they falter when faced with compositional reasoning tasks. Our research embarks on a quest to uncover the root causes of compositional reasoning failures of LLMs, uncovering that most of them stem from the improperly generated or leveraged implicit reasoning results. Inspired by our empirical findings, we resort to Logit Lens and an intervention experiment to dissect the inner hidden states of LLMs. This deep dive reveals that implicit reasoning results indeed surface within middle layers and play a causative role in shaping the final explicit reasoning results. Our exploration further locates multi-head self-attention (MHSA) modules within these layers, which emerge as the linchpins in accurate generation and leveraing of implicit reasoning results. Grounded on the above findings, we develop CREME, a lightweight method to patch errors in compositional reasoning via editing the located MHSA modules. Our empirical evidence stands testament to CREME's effectiveness, paving the way for autonomously and continuously enhancing compositional reasoning capabilities in language models.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
23 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Measuring and Narrowing the Compositionality Gap in Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Locating and Editing Factual Associations in GPT cites Fast Model Editing at Scale
- Causal Prompting: Debiasing Large Language Model Prompting Based on Front-Door Adjustment cites Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
- Causal Prompting: Debiasing Large Language Model Prompting Based on Front-Door Adjustment cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- How do Transformers Learn Implicit Reasoning? cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Least-to-Most Prompting Enables Complex Reasoning in Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Superficial Self-Improved Reasoners Benefit from Model Merging cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models cites Locating and Editing Factual Associations in GPT
- Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Mechanistic Detection and Mitigation of Hallucination in Large Reasoning Models cites Measuring and Narrowing the Compositionality Gap in Language Models
Cited by top-tier papers23
- Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language ModelsAsma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon et al.ICML 2024 · 197 citations
- Causal Prompting: Debiasing Large Language Model Prompting Based on Front-Door AdjustmentCongzhi Zhang, Linhai Zhang, Jialong Wu, Yulan He et al.AAAI 2025 · 42 citations
- Mechanistic Detection and Mitigation of Hallucination in Large Reasoning ModelsZhongxiang Sun, Qipeng Wang, Haoyu Wang, Xiao Zhang et al.ICLR 2026 · 30 citations
- How do Transformers Learn Implicit Reasoning?Jiaran Ye, Zijun Yao, Zhidian Huang, Liangming Pan et al.NeurIPS 2025 · 17 citations
- Superficial Self-Improved Reasoners Benefit from Model MergingXiangchi Yuan, Chunhui Zhang, Zheyuan Liu, Dachuan Shi et al.EMNLP 2025 · 15 citations
Builds on15
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Locating and Editing Factual Associations in GPTKevin Meng, David Bau, Alex Andonian, Yonatan BelinkovNeurIPS 2022 · 3,415 citations
- Fast Model Editing at ScaleEric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn et al.ICLR 2022 · 527 citations
- Least-to-Most Prompting Enables Complex Reasoning in Large Language ModelsDenny Zhou, Nathanael Schärli, Le Hou, Jason Wei et al.ICLR 2023 · 318 citations
- Measuring and Narrowing the Compositionality Gap in Language ModelsOfir Press, Muru Zhang, Sewon Min, Ludwig Schmidt et al.EMNLP Findings 2023 · 296 citations
Related papers
- Scaling Reasoning Hop Exposes Weaknesses: Demystifying and Improving Hop Generalization in Large Language ModelsZhaoyi Li, Jiatong Li, Gangwei Jiang, Linqi Song et al.ICLR 2026 · 1 citation
- Memory or Reasoning? Explore How LLMs Compute Mixed Arithmetic ExpressionsChengzhi Li, Heyan Huang, Ping Jian, Zhen Yang et al.ACL Findings 2025
- Faith and Fate: Limits of Transformers on CompositionalityNouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li et al.NeurIPS 2023 · 728 citations
- MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language ModelsDeepak Nathani, David Wang, Liangming Pan, William Yang WangEMNLP 2023 · 6 citations
- A Implies B: Circuit Analysis in LLMs for Propositional Logical ReasoningGuanzhe Hong, Nishanth Dikkala, Enming Luo, Cyrus Rashtchian et al.NeurIPS 2025 · 17 citations
