Lune

ACL Findings2025

BayesKD: Bayesian Knowledge Distillation for Compact LLMs in Constrained Fine-tuning Scenarios

Wei Li, Lujun Li, Mark G. Lee, Shengjie Sun, Lei Zhang, Wei Xue, Yike Guo

Abstract

Large language models (LLMs) have revolutionized various domains with their remarkable capabilities, but their massive parameter sizes pose significant challenges for finetuning and inference, especially in resourceconstrained environments. Conventional compression methods often result in substantial performance degradation within LLMs and struggle to restore model quality during fine-tuning. To address this challenge, we present Bayesian Knowledge Distillation (BayesKD), a novel distillation framework meticulously designed for compact LLMs in resource-constrained finetuning scenarios. Departing from conventional LLM distillation methods that introduce timeconsuming paradigms and fail to generalize in compressed LLM fine-tuning scenarios, our BayesKD develops the Logits Dual-Scaling, Knowledge Alignment Module, and Bayesian Distillation Optimization. In particular, our Logits Dual-Scaling strategy adaptively aligns the strength of the teacher's knowledge transfer, while the Knowledge Alignment Module bridges the gap between the teacher and student models by projecting their knowledge representations into a shared interval. Additionally, we employ Logits-Aware Bayesian Optimization to swiftly identify optimal settings based on these strategies, thereby enhancing model performance. Extensive experiments across diverse tasks demonstrate that BayesKD consistently outperforms baseline methods on various state-of-the-art LLMs, including LLaMA, Qwen2, Bloom, and Vicuna. Notably, our BayesKD achieves average accuracy gains of 2.99% and 4.05% over standard KD for the 8B parameter LLaMA and Qwen2 model. Codes are available in the supplementary materials.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TinyBERT: Distilling BERT for Natural Language Understanding cites DynaBERT: Dynamic BERT with Adaptive Width and Depth
  • DynaBERT: Dynamic BERT with Adaptive Width and Depth cites TinyBERT: Distilling BERT for Natural Language Understanding
  • MiniLLM: Knowledge Distillation of Large Language Models cites TinyBERT: Distilling BERT for Natural Language Understanding
  • MiniLLM: Knowledge Distillation of Large Language Models cites Improved Knowledge Distillation via Teacher Assistant
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites TinyBERT: Distilling BERT for Natural Language Understanding
  • LLM-Pruner: On the Structural Pruning of Large Language Models cites DynaBERT: Dynamic BERT with Adaptive Width and Depth
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on11

  1. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  2. Improved Knowledge Distillation via Teacher AssistantSeyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine et al.AAAI 2020 · 1,361 citations
  3. LLM-Pruner: On the Structural Pruning of Large Language ModelsXinyin Ma, Gongfan Fang, Xinchao WangNeurIPS 2023 · 994 citations
  4. DynaBERT: Dynamic BERT with Adaptive Width and DepthLu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang et al.NeurIPS 2020 · 401 citations
  5. MiniLLM: Knowledge Distillation of Large Language ModelsYuxian Gu, Li Dong, Furu Wei, Minlie HuangICLR 2024 · 95 citations

Related papers

  1. Dual-Space Knowledge Distillation for Large Language ModelsSongming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen et al.EMNLP 2024 · 3 citations
  2. LLaVA-KD: A Framework of Distilling Multimodal Large Language ModelsYuxuan Cai, Jiangning Zhang, Haoyang He, Xinwei He et al.ICCV 2025 · 9 citations
  3. PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference PairsRongzhi Zhang, Jiaming Shen, Tianqi Liu, Haorui Wang et al.ACL Findings 2024 · 1 citation
  4. daDPO: Distribution-Aware DPO for Distilling Conversational AbilitiesZhengze Zhang, Shiqi Wang, Yiqun Shen, Simin Guo et al.ACL Findings 2025 · 1 citation
  5. Distillation of Large Language Models via Concrete Score MatchingYeongmin Kim, Donghyeok Shin, Mina Kang, Byeonghu Na et al.ICLR 2026 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines