To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
Zayne Rea Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, Greg Durrett
Abstract
Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs). But for what kinds of tasks is this extra "thinking" really helpful? To analyze this, we conducted a quantitative meta-analysis covering over 100 papers using CoT and ran our own evaluations of 20 datasets across 14 models. Our results show that CoT gives strong performance benefits primarily on tasks involving math or logic, with much smaller gains on other types of tasks. On MMLU, directly generating the answer without CoT leads to almost identical accuracy as CoT unless the question or model's response contains an equals sign, indicating symbolic operations and reasoning. Following this finding, we analyze the behavior of CoT on these problems by separating planning and execution and comparing against tool-augmented LLMs. Much of CoT's gain comes from improving symbolic execution, but it underperforms relative to using a symbolic solver. Our results indicate that CoT can be applied selectively, maintaining performance while saving inference costs. Furthermore, they suggest a need to move beyond prompt-based CoT to new paradigms that better leverage intermediate computation across the whole range of LLM applications 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
85 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- CoT-Valve: Length-Compressible Chain-of-Thought Tuning cites Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- CoT-Valve: Length-Compressible Chain-of-Thought Tuning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Language Models are Few-Shot Learners
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
- Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens cites CoT-Valve: Length-Compressible Chain-of-Thought Tuning
- Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens cites Measuring Massive Multitask Language Understanding
- Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens cites Language Models are Few-Shot Learners
- Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Large Language Models are Zero-Shot Reasoners cites Language Models are Few-Shot Learners
- Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs cites Language Models are Few-Shot Learners
- When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Reasoning Models Better Express Their Confidence cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors cites Measuring Massive Multitask Language Understanding
- SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors cites Language Models are Few-Shot Learners
- SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers85
- CoT-Valve: Length-Compressible Chain-of-Thought TuningXinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang et al.ACL 2025 · 162 citations
- Reasoning Models Better Express Their ConfidenceDongkeun Yoon, Seungone Kim, Sohee Yang, Sunkyoung Kim et al.NeurIPS 2025 · 77 citations
- When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMsXiaomin Li, Zhou Yu, Zhiwei Zhang, Xupeng Chen et al.NeurIPS 2025 · 63 citations
- SimBench: Benchmarking the Ability of Large Language Models to Simulate Human BehaviorsTiancheng Hu, Joachim Baumann, Lorenzo Lupo, Nigel Collier et al.ICLR 2026 · 61 citations
- Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic LensXixian Yong, Xiao Zhou, Yingying Zhang, Jinlin Li et al.NeurIPS 2025 · 44 citations
Builds on48
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
Related papers
- Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What MattersBoshi Wang, Sewon Min, Xiang Deng, Jiaming Shen et al.ACL 2023 · 100 citations
- Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot DoZhuoran Jin, Kejian Zhu, Hongbang Yuan, Yupu Hao et al.ACL 2026
- Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans WorseRyan Liu, Jiayi Geng, Addison J. Wu, Ilia Sucholutsky et al.ICML 2025 · 4 citations
- The Impact of Reasoning Step Length on Large Language ModelsMingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao et al.ACL Findings 2024 · 39 citations
- Towards Revealing the Mystery behind Chain of Thought: A Theoretical PerspectiveGuhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye et al.NeurIPS 2023 · 470 citations
