Lune

ACL Findings2024

AS-ES Learning: Towards efficient CoT learning in small models

Nuwa Xi, Yuhan Chen, Sendong Zhao, Haochun Wang, Haochun Wang, Bing Qin, Ting Liu

Abstract

Chain-of-Thought (CoT) serves as a critical emerging ability in LLMs, especially when it comes to logical reasoning. Attempts have been made to induce such ability in small models as well by distilling from the data with CoT generated by Large Language Models (LLMs). However, existing methods often simply generate and incorporate more data from LLMs and fail to note the importance of efficiently utilizing existing CoT data. We here propose a new training paradigm AS-ES (Abstractive Segments -Extractive Segments) learning, which exploits the inherent information in CoT for iterative generation. Experiments show that our methods surpass the direct seq2seq training on CoT-extensive tasks like MWP and PET summarization, without data augmentation or altering the model itself. Furthermore, we explore the reason behind the inefficiency of small models in learning CoT and provide an explanation of why AS-ES learning works, giving insights into the underlying mechanism of CoT.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Self-Consistency Improves Chain of Thought Reasoning in Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Reasoning Scaffolding: Distilling the Flow of Thought from LLMs cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Reasoning Scaffolding: Distilling the Flow of Thought from LLMs cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Specializing Smaller Language Models towards Multi-Step Reasoning cites Self-Consistency Improves Chain of Thought Reasoning in Language Models
  • Specializing Smaller Language Models towards Multi-Step Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Reasoning Scaffolding: Distilling the Flow of Thought from LLMsXiangyu Wen, Junhua Huang, Zeju Li, Min Li et al.ICLR 2026 · 7 citations

Builds on14

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
  3. Specializing Smaller Language Models towards Multi-Step ReasoningYao Fu, Hao Peng, Litu Ou, Ashish Sabharwal et al.ICML 2023 · 347 citations
  4. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model SizesCheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost et al.ACL Findings 2023 · 317 citations
  5. RNNLogic: Learning Logic Rules for Reasoning on Knowledge GraphsMeng Qu, Jun-Kun Chen, Louis-Pascal A. C. Xhonneux, Yoshua Bengio et al.ICLR 2021 · 230 citations

Related papers

  1. Skip-Thinking: Chunk-wise Chain-of-Thought Distillation Enable Smaller Language Models to Reason Better and FasterXiaoshu Chen, Sihang Zhou, Ke Liang, Xiaoyu Sun et al.EMNLP 2025
  2. Rethinking Chain-of-Thought from the Perspective of Self-TrainingZongqian Wu, Baoduo Xu, Ruochen Cui, Mengmeng Zhan et al.ICML 2025
  3. UniCoTT: A Unified Framework for Structural Chain-of-Thought DistillationXianwei Zhuang, Zhihong Zhu, Zhichang Wang, Xuxin Cheng et al.ICLR 2025
  4. Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding?Renbiao Liu, Anqi Li, Chaoding Yang, Hui Sun et al.ICML 2025
  5. Investigating Mysteries of CoT-Augmented DistillationSomin Wadhwa, Silvio Amir, Byron C. WallaceEMNLP 2024 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines