FedEMoE: Improving Personalization on Heterogeneous Federated Learning via Elastic Mixture of Experts Architecture

Heterogeneous federated learning (HtFL) has emerged as a promising approach to address heterogeneity in local computational resources and data distribution. However, existing methods cause performance degradation of model personalization because personalized and generalized knowledge are either intertwined or dominated by one of them. To address this issue, we propose a novel Elastic Mixture of Experts (EMoE) architecture on HtFL, namely FedEMoE, decoupling personalization from generalization. Specially, FedEMoE employs a multi-scale feature extraction mechanism via personalized experts to enrich personalized knowledge. Furthermore, we design an elastic shared expert to break the transferred knowledge bottleneck across heterogeneous client models. The elastic shared expert can adaptively expand or shrink according to the status of each expert by the weight spectrum analysis, respectively. Extensive experiments across statistical and model heterogeneity settings demonstrate that FedEMoE significantly outperforms state-of-theart methods on the accuracy of each heterogeneous model over diverse datasets.

Building on this insight, our main contributions are:

• To the best of our knowledge, we first design a novel elastic MoE architecture for HtFL to decouple personalization from generalization, namely FedEMoE, ensuring local models never compromise their specialization for global consensus.

• We introduce a multi-scale feature extraction and knowledge exchange mechanism based on a MoE architecture at client-side. This mechanism maximizes the feature captured from local unique data and retained the ability to exchange knowledge with other clients, resulting in the improvement of personalization.

• We design an elastic MoE architecture where the shared expert acts as a dynamic repository of collective intelligence at server-side. Its structure can adaptively strengthen its representation capacity by preserving and integrating specialized knowledge rather than averaging it away.

• We evaluate FedEMoE in settings with model and statistical heterogeneity. Our extensive experiments and ablation studies demonstrate that FedEMoE is superior to state-of-the-art methods, improving task accuracy by up to 48.05%.