EFSkip: A New Error Feedback with Linear Speedup for Compressed Federated Learning with Arbitrary Data Heterogeneity

Due to the communication bottleneck in distributed and decentralized federated learning applications, algorithms using compressed communication have attracted significant attention. The Error Feedback (EF) is a widely-studied compression framework for convergence with biased compressors such as top-k sparsification. Although various improvements have been obtained in recent years, the theoretical guarantee for EF-type framework is still limited. Previous works either 1) rely on strong assumptions such as bounded gradient/dissimilarity assumptions, thus can not deal with arbitrary data heterogeneity and also slow the convergence speed, or 2) can not enjoy linear speedup in the number of clients. In this work, we propose a new EFSkip framework which removes the strong assumptions to allow arbitrary data heterogeneity and enjoys linear speedup for significantly improving upon previous results. In particular, EFSkip achieves the complexity result O( σ 2 nϵ 4 + 1 ϵ 2 ) while previous EF21 only obtains O( σ 2 δ 3 ϵ 4 + 1 δϵ 2 ), i.e., EFSkip enjoys the linear speedup in the number of clients n (reducing the result linearly using more clients) and also removes the compression factor δ (matching the result without compression). We also show that EF-Skip enjoys linear speedup and achieves faster convergence for nonconvex problems satisfying Polyak-Łojasiewicz (PL) condition. We believe that the new EFSkip framework will have a large impact on the communication-and computationefficient distributed and decentralized federated learning. * The work of Hongyan Bao was conducted during her stay in Zhize Li's group at SMU.