ACL Findings2024
Learning to Maximize Mutual Information for Chain-of-Thought Distillation
Xin Chen, Hanxian Huang, Yanjun Gao, Yi Wang, Jishen Zhao, Ke Ding
Abstract
Knowledge distillation, the technique of transferring knowledge from large, complex models to smaller ones, marks a pivotal step towards efficient AI deployment. Distilling Step-by-Step (DSS), a novel method utilizing chain-of-thought (CoT) distillation, has demonstrated promise by imbuing smaller models with the superior reasoning capabilities of their larger counterparts.In DSS, the distilled model acquires the ability to generate rationales and predict labels concurrently through a multi-task learning framework.However, DSS overlooks the intrinsic relationship between the two training tasks, leading to ineffective integration of CoT knowledge with the task of label prediction.To this end, we investigate the mutual relationship of the two tasks from Information Bottleneck perspective and formulate it as maximizing the mutual information of the representation features of the two tasks.We propose a variational approach to solve this optimization problem using a learning-based method.Our experimental results across four datasets demonstrate that our method outperforms the state-of-the-art DSS.Our findings offer insightful guidance for future research on language model distillation as well as applications involving CoT.Codes are available at https://github.com/xinchen9/ cot_distillation_ACL2024
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation cites TinyBERT: Distilling BERT for Natural Language Understanding
- Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation cites Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes cites Adversarial NLI: A New Benchmark for Natural Language Understanding
- FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language Model cites On Mutual Information Maximization for Representation Learning
Cited by top-tier papers7
- Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM ReasoningChen Qian, Dongrui Liu, Haochen Wen, Zhen Bai et al.NeurIPS 2025 · 63 citations
- Activation-Guided Consensus Merging for Large Language ModelsYuxuan Yao, Shuqi Liu, Zehua Liu, Qintong Li et al.NeurIPS 2025 · 14 citations
- FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language ModelJinwei Hu, Zhenglin Huang, Xiangyu Yin, Wenjie Ruan et al.NeurIPS 2025 · 3 citations
- Pedagogically-Inspired Data Synthesis for Language Model Knowledge DistillationBowei He, Yankai Chen, Xiaokun Zhang, Linghe Kong et al.ICLR 2026 · 2 citations
- Predicting Through Generation: Why Generation Is Better for PredictionMd. Kowsher, Nusrat Jahan Prottasha, Prakash Bhat, Chun-Nam Yu et al.ACL 2025
Builds on23
- TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
- Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations
- On Mutual Information Maximization for Representation LearningMichael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly et al.ICLR 2020 · 559 citations
- Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model SizesCheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost et al.ACL Findings 2023 · 317 citations
- Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion ModelsWeijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun et al.NeurIPS 2023 · 268 citations
Related papers
- Improving Reasoning Capabilities in Small Models through Mixture-of-layers Distillation with Stepwise Attention on Key InformationYao Chen, Jiawei Sheng, Wenyuan Zhang, Tingwen LiuEMNLP 2025
- Mentor-KD: Making Small Language Models Better Multi-step ReasonersHojae Lee, Junho Kim, SangKeun LeeEMNLP 2024
- MCC-KD: Multi-CoT Consistent Knowledge DistillationHongzhan Chen, Siyue Wu, Xiaojun Quan, Rui Wang et al.EMNLP Findings 2023 · 14 citations
- Towards Efficient CoT Distillation: Self-Guided Rationale Selector for Better Performance with Fewer RationalesJianzhi Yan, Le Liu, Youcheng Pan, Shiwei Chen et al.EMNLP Findings 2025
- Mixed Distillation Helps Smaller Language Models Reason BetterChenglin Li, Qianglong Chen, Liangyue Li, Caiyu Wang et al.EMNLP Findings 2024 · 3 citations
