EMNLP Findings2024
Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning
Abstract
Large language models (LLMs) have been shown to perform better when asked to reason step-by-step before answering a question. However, it is unclear to what degree the model's final answer is faithful to the stated reasoning steps. In this paper, we perform a causal mediation analysis on twelve LLMs to examine how intermediate reasoning steps generated by the LLM influence the final outcome and find that LLMs do not reliably use their intermediate reasoning steps when generating an answer. To address this issue, we introduce FRODO, a framework to tailor small-sized LMs to generate correct reasoning steps and robustly reason over these steps. FRODO consists of an inference module that learns to generate correct reasoning steps using an implicit causal reward function and a reasoning module that learns to faithfully reason over these intermediate inferences using a counterfactual and causal preference objective. Our experiments show that FRODO significantly outperforms four competitive baselines. Furthermore, FRODO improves the robustness and generalization ability of the reasoning LM, yielding higher performance on out-of-distribution test sets. Finally, we find that FRODO's rationales are more faithful to its final answer predictions than standard supervised fine-tuning.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
31 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future cites Reflexion: language agents with verbal reinforcement learning
- Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future cites Large Language Models are Zero-Shot Reasoners
- Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in Explanations cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning cites Large Language Models are Zero-Shot Reasoners
- FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring cites Large Language Models are Zero-Shot Reasoners
- CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Reflexion: language agents with verbal reinforcement learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- When Thinking Drifts: Evidential Grounding for Robust Video Reasoning cites Large Language Models are Zero-Shot Reasoners
- When Thinking Drifts: Evidential Grounding for Robust Video Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers31
- CoT Red-Handed: Stress Testing Chain-of-Thought MonitoringBenjamin Arnav, Pablo Bernabeu-Perez, Nathan Helm-Burger, Timothy H. Kostolansky et al.NeurIPS 2025 · 50 citations
- Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and FutureZheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu et al.ACL 2024 · 36 citations
- FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought ReasoningXu Shen, Song Wang, Zhen Tan, Laura Yao et al.ICLR 2026 · 28 citations
- When Thinking Drifts: Evidential Grounding for Robust Video ReasoningRomy Luo, Zihui Xue, Alex Dimakis, Kristen GraumanNeurIPS 2025 · 21 citations
- Truthful or Fabricated? Using Causal Attribution to Mitigate Reward Hacking in ExplanationsPedro Lobato Ferreira, Wilker Aziz, Ivan TitovICLR 2026 · 12 citations
Builds on29
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- Supervised Contrastive LearningPrannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna et al.NeurIPS 2020 · 7,049 citations
- Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
Related papers
- Selection-Inference: Exploiting Large Language Models for Interpretable Logical ReasoningAntonia Creswell, Murray Shanahan, Irina HigginsICLR 2023 · 110 citations
- Faithful Question Answering with Monte-Carlo PlanningRuixin Hong, Hongming Zhang, Hong Zhao, Dong Yu et al.ACL 2023 · 7 citations
- Reasoning Elicitation in Language Models via Counterfactual FeedbackAlihan Hüyük, Xinnuo Xu, Jacqueline R. M. A. Maasch, Aditya V. Nori et al.ICLR 2025
- Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision StudyYujun Zhou, Jiayi Ye, Zipeng Ling, Yufei Han et al.EMNLP Findings 2025 · 3 citations
- SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-ThoughtGuanghao Li, Wenhao Jiang, Mingfeng Chen, Yan Li et al.NeurIPS 2025 · 8 citations
