EMNLP Findings2024
Divide-or-Conquer? Which Part Should You Distill Your LLM?
Zhuofeng Wu, Richard He Bai, Aonan Zhang, Jiatao Gu, V. G. Vinod Vydiswaran, Navdeep Jaitly, Yizhe Zhang
Abstract
Recent methods have demonstrated that Large Language Models (LLMs) can solve reasoning tasks better when they are encouraged to solve subtasks of the main task first. In this paper we devise a similar strategy that breaks down reasoning tasks into a problem decomposition phase and a problem solving phase and show that the strategy is able to outperform a single stage solution. Further, we hypothesize that the decomposition should be easier to distill into a smaller model compared to the problem solving because the latter requires large amounts of domain knowledge while the former only requires learning general problem solving strategies. We propose methods to distill these two capabilities and evaluate their impact on reasoning outcomes and inference cost. We find that we can distill the problem decomposition phase and at the same time achieve good generalization across tasks, datasets, and models. However, it is harder to distill the problem solving capability without losing performance and the resulting distilled model struggles with generalization. These results indicate that by using smaller, distilled problem decomposition models in combination with problem solving LLMs we can achieve reasoning with cost-efficient inference and local adaptation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Language Models are Few-Shot Learners
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQL cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- Probing to Refine: Reinforcement Distillation of LLM Reasoners via Explanatory Inversion cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Probing to Refine: Reinforcement Distillation of LLM Reasoners via Explanatory Inversion cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Minimal Free Resolution Guided Adaptive Tree Reasoning cites Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
- Training Software Engineering Agents and Verifiers with SWE-Gym cites Training language models to follow instructions with human feedback
Cited by top-tier papers6
- SQLens: An End-to-End Framework for Error Detection and Correction in Text-to-SQLYue Gong, Chuan Lei, Xiao Qin, Kapil Vaidya et al.NeurIPS 2025 · 21 citations
- Probing to Refine: Reinforcement Distillation of LLM Reasoners via Explanatory InversionZhen Tan, Chengshuai Zhao, Song Wang, Jundong Li et al.ICLR 2026
- Plug-and-Play Clarifier: A Zero-Shot Multimodal Framework for Egocentric Intent DisambiguationSicheng Yang, Yukai Huang, Weitong Cai, Shitong Sun et al.AAAI 2026
- Minimal Free Resolution Guided Adaptive Tree ReasoningDezhao Tang, Meihan Liu, Yulai Tong, Guan Yuan et al.ACL 2026
- Training Software Engineering Agents and Verifiers with SWE-GymJiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly et al.ICML 2025
Builds on22
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
Related papers
- Teach Small Models to Reason by Curriculum DistillationWangyi Jiang, Yaojie Lu, Hongyu Lin, Xianpei Han et al.EMNLP 2025
- Distilling LLM Agent into Small Models with Retrieval and Code ToolsMinki Kang, Jongwon Jeong, Seanie Lee, Jaewoong Cho et al.NeurIPS 2025 · 51 citations
- Distilling Reasoning Capabilities into Smaller Language ModelsKumar Shridhar, Alessandro Stolfo, Mrinmaya SachanACL Findings 2023 · 80 citations
- Disentangling Memory and Reasoning Ability in Large Language ModelsMingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang et al.ACL 2025
- Improve Student's Reasoning Generalizability through Cascading Decomposed CoTs DistillationChengwei Dai, Kun Li, Wei Zhou, Songlin HuEMNLP 2024 · 4 citations
