FedEcover: Fast and Stable Converging Model-Heterogeneous Federated Learning with Efficient-Coverage Submodel Extraction

Federated learning (FL) has achieved favorable progress in addressing the data silo problem without compromising clients' data privacy. In real-world scenarios, there are numerous low-capacity clients, i.e., devices with limited resources like computational power, storage and bandwidth, holding unique and valuable data. Yet the conventional model-homogeneous paradigm is unsuitable due to its uniform model demands on all clients. To effectively utilize the data from clients of various capacities for learning a well-performing global model, researchers have proposed submodel extraction-based partial training methods allowing clients to locally train heterogeneous submodels of different sizes. However, existing partial training methods are inadequate in terms of parameter space coverage efficiency and convergence stability, which adversely affects convergence rate and the final performance. In this work, we introduce FedEcover, a model-heterogeneous framework to learn a fast and stable converging global model in challenging scenarios with dual heterogeneity of data and client capacity. Specifically, our framework incorporates an efficient submodel extraction scheme applying a random sampling without replacement strategy and a step-size decay mechanism in the global aggregation process, to enable the global model fully leveraging the heterogeneous data distributed across capacity-heterogeneous clients. Experimental results on multiple models and datasets demonstrate that our framework outperforms existing submodel extraction-based partial training methods and model-homogeneous FedAvg in both convergence rate and converged performance of the global model.