B-ary Tree Push-Pull Method is Provably Efficient for Distributed Learning on Heterogeneous Data

This paper considers the distributed learning problem where a group of agents cooperatively minimizes the summation of their local cost functions based on peer-to-peer communication. Particularly, we propose a highly efficient algorithm, termed "B-ary Tree Push-Pull" (BTPP), that employs two B-ary spanning trees for distributing the information related to the parameters and stochastic gradients across the network. The simple method is efficient in communication since each agent interacts with at most (B + 1) neighbors per iteration. More importantly, BTPP achieves linear speedup for smooth nonconvex and strongly convex objective functions with only Õ(n) and Õ(1) transient iterations, respectively, significantly outperforming the state-of-the-art results to the best of our knowledge. Our code is available at https://github.com/ryou98/BTPP . ALGORITHM PER-ITER COMM. SIZE n BASED GRAPH TRANS. ITER. 1) ARBITRARY 2 Õ(n) ALGORITHM PER-ITER COMM. SIZE n BASED GRAPH TRANS. ITER. DSGD (RING) [18] Θ(1) ARBITRARY 1 Õ(n 5 ) STATIC EXP. [34] Θ(ln(n)) ARBITRARY 1 Õ(n) O.-P. EXP. [34] 1 POWER OF 2 Θ(ln(n)) Õ(n) RELAYSGD [31] Θ(1)