EMNLP Findings2022
Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization
Aref Jafari, Ivan Kobyzev, Mehdi Rezagholizadeh, Pascal Poupart, Ali Ghodsi
Abstract
Knowledge Distillation (KD) has been extensively used for natural language understanding (NLU) tasks to improve a small model's (a student) generalization by transferring the knowledge from a larger model (a teacher). Although KD methods achieve state-of-the-art performance in numerous settings, they suffer from several problems limiting their performance. It is shown in the literature that the capacity gap between the teacher and the student networks can make KD ineffective. Additionally, existing KD techniques do not mitigate the noise in the teacher's output: modeling the noisy behaviour of the teacher can distract the student from learning more useful features. We propose a new KD method that addresses these problems and facilitates the training compared to previous techniques. Inspired by continuation optimization, we design a training procedure that optimizes the highly non-convex KD objective by starting with the smoothed version of this objective and making it more complex as the training proceeds.
Our method (Continuation-KD) achieves stateof-the-art performance across various compact architectures on NLU (GLUE benchmark) and computer vision tasks (CIFAR-10 and CIFAR-100).
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
- A Good Learner can Teach Better: Teacher-Student Collaborative Knowledge Distillation cites TinyBERT: Distilling BERT for Natural Language Understanding
- A Good Learner can Teach Better: Teacher-Student Collaborative Knowledge Distillation cites Improved Knowledge Distillation via Teacher Assistant
- A Good Learner can Teach Better: Teacher-Student Collaborative Knowledge Distillation cites Language Models are Few-Shot Learners
Cited by top-tier papers1
Builds on6
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
- Improved Knowledge Distillation via Teacher AssistantSeyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine et al.AAAI 2020 · 1,361 citations
- Online Knowledge Distillation via Collaborative LearningQiushan Guo, Xinjiang Wang, Yichao Wu, Zhipeng Yu et al.CVPR 2020
Related papers
- How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language UnderstandingTianda Li, Ahmad Rashid, Aref Jafari, Pranav Sharma et al.EMNLP Findings 2021 · 6 citations
- On the Generalization vs Fidelity Paradox in Knowledge DistillationSuhas Kamasetty Ramesh, Ayan Sengupta, Tanmoy ChakrabortyACL Findings 2025 · 1 citation
- A Good Teacher Adapts Their Knowledge for DistillationChengyao Qian, Trung Le, Mehrtash HarandiICCV 2025 · 8 citations
- Towards Zero-Shot Knowledge Distillation for Natural Language ProcessingAhmad Rashid, Vasileios Lioutas, Abbas Ghaddar, Mehdi RezagholizadehEMNLP 2021 · 25 citations
- Co-training and Co-distillation for Quality Improvement and Compression of Language ModelsHayeon Lee, Rui Hou, Jongpil Kim, Davis Liang et al.EMNLP Findings 2023 · 1 citation
