Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models

Knowledge distillation establishes a learning paradigm that learns from both data supervision and teacher guidance. However, the optimal balance between learning from data and learning from the teacher is hard to determine, as some samples are data-noisy while others are teacher-uncertain. This raises a pressing need to adaptively balance data and teacher supervision. We propose Beta-weighted Knowledge Distillation (Beta-KD), an uncertainty-aware distillation framework that adaptively modulates how much the student rely on the teacher guidence. Specifically, we formulate teacher-student learning from a unified Bayesian perspective and interpret teacher supervision as a Gibbs prior over student activations. This yields a closed-form, uncertainty-aware weighting mechanism and supports arbitrary distillation objectives and combination. Extensive experiments are conducted on multimodal VQA benchmarks by distilling a student Vision-Language Model (MobiVLM and LLaVA) from a large teacher VLM. The results demonstrate that Beta-KD consistently outperforms existing knowledge distillation methods. Code is available at https: //github.com/Jingchensun/beta-kd.