DKDR: Dynamic Knowledge Distillation for Reliability in Federated Learning

Federated Learning (FL) has demonstrated a promising future in privacy-friendly collaboration but it faces the data heterogeneity problem. Knowledge Distillation (KD) can serve as an effective method to address this issue. However, challenges arise from the unreliability of existing distillation methods in multi-domain scenarios. Prevalent distillation solutions primarily aim to fit the distributions of the global model directly by minimizing forward Kullback-Leibler divergence (KLD). This results in significant bias when the outputs of the global model are multi-peaked, which indicates the unreliability of distillation pathway. Meanwhile, cross-domain update conflicts can notably reduce the accuracy of the global model (teacher model) in certain domains, reflecting the unreliability of the teacher model in these domains. In this work, we propose DKDR (Dynamic Knowledge Distillation for Reliability in Federated Learning), which dynamically assigns weights to forward and reverse KLD based on knowledge discrepancies. This enables clients to fit the outputs from the teacher precisely. Moreover, we use knowledge decoupling to identify domain experts, thus clients can acquire reliable domain knowledge from experts. Empirical results from single-domain and multi-domain image classification tasks demonstrate the effectiveness of the proposed method and the efficiency of its key modules. The code is available at https://github.com/YueyangYuan/DKDR.

Federated learning is a collaborative paradigm [21,60,27,[14][15][16]62], enabling multiple clients to jointly train a shared global model [39,28,15] while ensuring privacy protection [52]. However, the distributed data is collected from different sources with diverse preferences and brings the nonindependent and identically distributed (non-IID) characteristics. Knowledge distillation [13,4] addresses this challenge effectively by aligning the outputs of local models with the global model. It brings the optimization objectives of each client closer together thus resolving the problem. However, existing distillation methods [25,11,36,5] typically use forward KLD to fit the distributions of the global model. We argue that this approach is unreliable in multi-domain scenarios. Given the global model distribution Z(y|x) and the local model distribution Z w (y|x) parameterized by w, standard knowledge distillation objectives aim to minimize the forward KLD between them, denoted as KL[Z|Z w ]. This approach compels Z w to encompass all modes of Z. However, we † Equal Contribution. * Corresponding Author. 39th Conference on Neural Information Processing Systems (NeurIPS 2025). (b) Passway Unreliability Global Distribution Forward KLD Entropy Multi-peaked Global Distribution Forward KLD Significant Bias ！ (c) Teacher Unreliability Accuracy Global model is less effective in certain domains! Client Model Global Model Accuracy Webcam Amazon Dslr 10.8 36.2 43.3 (a) Typical Distillation Methods Global Model (Teacher Model) Client Models Entropy Accuracy 0.41 83.7 Webcam 0.45 72.0 Amazon 0.32 77.0 Dslr Aggregation Low Entropy single-peaked distribution