ACL Findings2023
Cost-effective Distillation of Large Language Models
Sayantan Dasgupta, Trevor Cohn, Timothy Baldwin
Abstract
Knowledge distillation (KD) involves training a small "student" model to replicate the strong performance of a high-capacity "teacher" model, enabling efficient deployment in resource-constrained settings. Topperforming methods tend to be task-or architecture-specific and lack generalizability. Several existing approaches require pretraining of the teacher on task-specific datasets, which can be costly for large and unstable for small datasets. Here we propose an approach for improving KD through a novel distillation loss agnostic to the task and model architecture. We successfully apply our method to the distillation of the BERT-base and achieve highly competitive results from the distilled student across a range of GLUE tasks, especially for tasks with smaller datasets. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- TinyBERT: Distilling BERT for Natural Language Understanding cites MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices cites TinyBERT: Distilling BERT for Natural Language Understanding
- Don't Ignore the Tail: Decoupled Distillation Produces Top Maths Students on an Academic Budget cites Improving Language Model Distillation through Hidden State Matching
- Improving Language Model Distillation through Hidden State Matching cites TinyBERT: Distilling BERT for Natural Language Understanding
- Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention cites Fast Transformers with Clustered Attention
- Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention cites Language Models are Few-Shot Learners
- Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression cites Language Models are Few-Shot Learners
Cited by top-tier papers5
- Backdoor Collapse: Eliminating Unknown Threats Via Known Backdoor Aggregation In Language ModelsLiang Lin, Miao Yu, Moayad Aloqaily, Zhenhong Zhou et al.ACL 2026 · 4 citations
- Surgical Feature-Space Decomposition of LLMs: Why, When and How?Arnav Chavan, Nahush Lele, Deepak K. GuptaACL 2024 · 1 citation
- Don't Ignore the Tail: Decoupled Distillation Produces Top Maths Students on an Academic BudgetSayantan Dasgupta, Trevor Cohn, Tim BaldwinICML 2026
- Improving Language Model Distillation through Hidden State MatchingSayantan Dasgupta, Trevor CohnICLR 2025
- Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM CompressionPeijie Dong, Zhenheng Tang, Xiang Liu, Lujun Li et al.ICML 2025
Builds on9
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
- Nyströmformer: A Nyström-based Algorithm for Approximating Self-AttentionYunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan et al.AAAI 2021 · 675 citations
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesZhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu et al.ACL 2020 · 660 citations
- Fast Transformers with Clustered AttentionApoorv Vyas, Angelos Katharopoulos, François FleuretNeurIPS 2020 · 193 citations
Related papers
- Maximizing the Effectiveness of Larger BERT Models for CompressionWen-Shu Fan, Su Lu, Shangyu Xing, Xin-Chun Li et al.ACL 2025
- Towards Efficient Pre-Trained Language Model via Feature Correlation DistillationKun Huang, Xin Guo, Meng WangNeurIPS 2023 · 8 citations
- HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained TransformersChen Liang, Haoming Jiang, Zheng Li, Xianfeng Tang et al.ICLR 2023 · 9 citations
- Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language ModelsDongkuan Xu, Subhabrata Mukherjee, Xiaodong Liu, Debadeepta Dey et al.NeurIPS 2022 · 21 citations
- MixKD: Towards Efficient Distillation of Large-scale Language ModelsKevin J. Liang, Weituo Hao, Dinghan Shen, Yufan Zhou et al.ICLR 2021 · 90 citations
