EMNLP Findings2021
RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation
Peng Lu, Abbas Ghaddar, Ahmad Rashid, Mehdi Rezagholizadeh, Ali Ghodsi, Philippe Langlais
Abstract
Knowledge Distillation (KD) is extensively used in Natural Language Processing to compress the pre-training and task-specific finetuning phases of large neural language models. A student model is trained to minimize a convex combination of the prediction loss over the labels and another over the teacher output. However, most existing works either fix the interpolating weight between the two losses apriori or vary the weight using heuristics. In this work, we propose a novel samplewise loss weighting method, RW-KD. A metalearner, simultaneously trained with the student, adaptively re-weights the two losses for each sample. We demonstrate, on 7 datasets of the GLUE benchmark, that RW-KD outperforms other loss re-weighting methods for KD.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- TinyBERT: Distilling BERT for Natural Language Understanding cites MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices cites TinyBERT: Distilling BERT for Natural Language Understanding
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers cites TinyBERT: Distilling BERT for Natural Language Understanding
- EA-KD: Entropy-Based Adaptive Knowledge Distillation cites TinyBERT: Distilling BERT for Natural Language Understanding
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation cites Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective
Cited by top-tier papers4
- EA-KD: Entropy-Based Adaptive Knowledge DistillationChi-Ping Su, Ching-Hsun Tseng, Bin Pu, Lei Zhao et al.ICCV 2025 · 5 citations
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge DistillationChengming Hu, Haolun Wu, Xuan Li, Chen Ma et al.ICLR 2024 · 4 citations
- Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting LayersYoumin Ko, Sungjong Seo, Hyunjoon KimNeurIPS 2025 · 2 citations
- BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMsAbbas Ghaddar, Ivan Kobyzev, Boxing Chen, Yufei CuiACL 2026
Builds on13
- TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesZhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu et al.ACL 2020 · 660 citations
- MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained TransformersWenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong et al.ACL Findings 2021 · 260 citations
- Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff PerspectiveHelong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou et al.ICLR 2021 · 209 citations
- Show, Attend and Distill: Knowledge Distillation via Attention-based Feature MatchingMingi Ji, Byeongho Heo, Sungrae ParkAAAI 2021 · 194 citations
Related papers
- Tutoring Helps Students Learn Better: Improving Knowledge Distillation for BERT with Tutor NetworkJunho Kim, Jun-Hyung Park, Mingyu Lee, Wing-Lam Mok et al.EMNLP 2022 · 4 citations
- Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across DomainsHaojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang et al.ACL 2021
- Reinforced Multi-Teacher Selection for Knowledge DistillationFei Yuan, Linjun Shou, Jian Pei, Wutao Lin et al.AAAI 2021 · 155 citations
- Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language ModelsDongkuan Xu, Subhabrata Mukherjee, Xiaodong Liu, Debadeepta Dey et al.NeurIPS 2022 · 21 citations
- How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language UnderstandingTianda Li, Ahmad Rashid, Aref Jafari, Pranav Sharma et al.EMNLP Findings 2021 · 6 citations
