Lune

EMNLP Findings2021

RW-KD: Sample-wise Loss Terms Re-Weighting for Knowledge Distillation

Peng Lu, Abbas Ghaddar, Ahmad Rashid, Mehdi Rezagholizadeh, Ali Ghodsi, Philippe Langlais

Abstract

Knowledge Distillation (KD) is extensively used in Natural Language Processing to compress the pre-training and task-specific finetuning phases of large neural language models. A student model is trained to minimize a convex combination of the prediction loss over the labels and another over the teacher output. However, most existing works either fix the interpolating weight between the two losses apriori or vary the weight using heuristics. In this work, we propose a novel samplewise loss weighting method, RW-KD. A metalearner, simultaneously trained with the student, adaptively re-weights the two losses for each sample. We demonstrate, on 7 datasets of the GLUE benchmark, that RW-KD outperforms other loss re-weighting methods for KD.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TinyBERT: Distilling BERT for Natural Language Understanding cites MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
  • MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices cites TinyBERT: Distilling BERT for Natural Language Understanding
  • MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers cites TinyBERT: Distilling BERT for Natural Language Understanding
  • EA-KD: Entropy-Based Adaptive Knowledge Distillation cites TinyBERT: Distilling BERT for Natural Language Understanding
  • Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation cites Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. EA-KD: Entropy-Based Adaptive Knowledge DistillationChi-Ping Su, Ching-Hsun Tseng, Bin Pu, Lei Zhao et al.ICCV 2025 · 5 citations
  2. Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge DistillationChengming Hu, Haolun Wu, Xuan Li, Chen Ma et al.ICLR 2024 · 4 citations
  3. Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting LayersYoumin Ko, Sungjong Seo, Hyunjoon KimNeurIPS 2025 · 2 citations
  4. BOSCH: Black-Box Binary Optimization for Short-Context Attention-Head Selection in LLMsAbbas Ghaddar, Ivan Kobyzev, Boxing Chen, Yufei CuiACL 2026

Builds on13

  1. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  2. MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesZhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu et al.ACL 2020 · 660 citations
  3. MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained TransformersWenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong et al.ACL Findings 2021 · 260 citations
  4. Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff PerspectiveHelong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou et al.ICLR 2021 · 209 citations
  5. Show, Attend and Distill: Knowledge Distillation via Attention-based Feature MatchingMingi Ji, Byeongho Heo, Sungrae ParkAAAI 2021 · 194 citations

Related papers

  1. Tutoring Helps Students Learn Better: Improving Knowledge Distillation for BERT with Tutor NetworkJunho Kim, Jun-Hyung Park, Mingyu Lee, Wing-Lam Mok et al.EMNLP 2022 · 4 citations
  2. Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across DomainsHaojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang et al.ACL 2021
  3. Reinforced Multi-Teacher Selection for Knowledge DistillationFei Yuan, Linjun Shou, Jian Pei, Wutao Lin et al.AAAI 2021 · 155 citations
  4. Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language ModelsDongkuan Xu, Subhabrata Mukherjee, Xiaodong Liu, Debadeepta Dey et al.NeurIPS 2022 · 21 citations
  5. How to Select One Among All ? An Empirical Study Towards the Robustness of Knowledge Distillation in Natural Language UnderstandingTianda Li, Ahmad Rashid, Aref Jafari, Pranav Sharma et al.EMNLP Findings 2021 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines