Lune

ACL Findings2026

Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation

Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahi Baarzi, Hejian Sang, Shao Tang, Qingquan Song, Zhipeng Wang, Muhammad Abdul-Mageed

Abstract

Distilling the capabilities from a large reasoning model (LRM) to a smaller student model often involves training on substantial amounts of reasoning data. However, knowledge distillation (KD) over lengthy sequences with prompt (P), chain-of-thought (CoT), and answer (A) sections makes the process computationally expensive. In this work, we investigate how the allocation of supervision across different sections (P, CoT, A) affects student performance. Our analysis shows that selective KD over only the CoT tokens can be effective when the prompt and answer information is encompassed by it. Building on this insight, we establish a truncation protocol to quantify computation-quality tradeoffs as a function of sequence length. We observe that beyond a specific length, longer training sequences provide marginal returns for downstream performance but require substantially higher memory and FLOPs. To this end, training on only the first 50% of tokens of every training sequence can retain, on average, ≈ 91% of full-sequence performance on math benchmarks while reducing training time, memory usage, and FLOPs by about 50% each. Codes are available at https://github.com /weiruichen01/distilling-the-essence.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Self-Consistency Improves Chain of Thought Reasoning in Language Models cites Large Language Models are Zero-Shot Reasoners
  • Large Language Models are Zero-Shot Reasoners cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Self-Consistency Improves Chain of Thought Reasoning in Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes cites Large Language Models are Zero-Shot Reasoners
  • Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Large Language Models are Zero-Shot Reasoners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes cites Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
  • On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on14

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Large Language Models are Zero-Shot ReasonersTakeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo et al.NeurIPS 2022 · 8,168 citations
  3. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
  4. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model SizesCheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost et al.ACL Findings 2023 · 317 citations
  5. On-Policy Distillation of Language Models: Learning from Self-Generated MistakesRishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk et al.ICLR 2024 · 311 citations

Related papers

  1. Investigating Mysteries of CoT-Augmented DistillationSomin Wadhwa, Silvio Amir, Byron C. WallaceEMNLP 2024 · 1 citation
  2. When Internalization Fails: Finding Better Targets for Reasoning CompressionMourad Heddaya, Manley Roberts, Rohan Wadhawan, Chenhao TanACL Findings 2026
  3. DRP: Distilled Reasoning Pruning with Mathematical Skill-aware Step Decomposition for Efficient Large Reasoning ModelsYuxuan Jiang, Dawei Li, Francis FerraroACL Findings 2026 · 2 citations
  4. Think Better, Not Longer: Token-Level Marginal Utility for Efficient Reasoning in Large Reasoning ModelsJiawei Li, Yang Gao, Huashan Sun, Chong FengACL 2026
  5. Pru-CoT: Towards Efficient Reasoning Distillation via Pruning Chain-of-ThoughtHan Liu, Shuotian Ma, Hui Li, Xiaotong Zhang et al.ACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines