Beyond Euclidean Gossip: KL-Barycentric Consensus on Heterogeneous and Imbalanced Images
Abstract
Fully decentralized deep learning removes global servers and ensures local data privacy. However, Euclidean consensus, averaging weights, gradients or momentum, may degrade under the situations with non-i.i.d. (nonindependent and identically distributed) data and client size imbalance. We propose a geometry-aware approach based on natural gradient variational inference. Clients communicate in the expectation parameter space of an exponential family, where simple linear mixing yields a forward Kullback-Leibler (KL) barycenter consensus. The aggregate is the model closest to all client distributions, aligning updates across heterogeneous sites and mitigating distribution shift. We further provide a lightweight decentralized Adam implementation where each client maintains a diagonal Gaussian posterior and operates entirely in the expectation space for both optimization and consensus. We prove convergence for convex losses on connected graphs. On CIFAR-100 and a medical image segmentation benchmark, our method substantially outperforms Euclidean-space consensus baselines under severe noni.i.d. and client-imbalance cases, achieving around 20% accuracy gain on CIFAR-100, while matching the communication budget and improving training stability. The code is available at https://github.com/x-lu/Beyond-euclidean-gossip.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on1
Related papers
- FedCDWA: Decoupled Federated Prototype Distillation with Hierarchical Wasserstein AggregationZhenshen Liu, Kai Fan, Wenjie Li, Kuan Zhang et al.ICML 2026
- Personalized Bayesian Federated Learning with Wasserstein Barycenter AggregationTing Wei, Biao Mei, Junliang Lyu, Renquan Zhang et al.NeurIPS 2025 · 3 citations
- Interaction-Aware Gaussian Weighting for Clustered Federated LearningAlessandro Licciardi, Davide Leo, Eros Fanì, Barbara Caputo et al.ICML 2025
- DecentLaM: Decentralized Momentum SGD for Large-batch Deep TrainingKun Yuan, Yiming Chen, Xinmeng Huang, Yingya Zhang et al.ICCV 2021 · 73 citations
- Elastic Aggregation for Federated OptimizationDengsheng Chen, Jie Hu, Vince Junkai Tan, Xiaoming Wei et al.CVPR 2023
