Lune

CVPR2022Top-tier venue

Federated Learning with Position-Aware Neurons

Xin-Chun Li, Yichu Xu, Shaoming Song, Bingshuai Li, Yinchuan Li, Yunfeng Shao, De-Chuan Zhan

Abstract

Federated Learning (FL) fuses collaborative models from local nodes without centralizing users' data. The permutation invariance property of neural networks and the non-i.i.d. data across clients make the locally updated parameters imprecisely aligned, disabling the coordinate-based parameter averaging. Traditional neurons do not explicitly consider position information. Hence, we propose Position-Aware Neurons (PANs) as an alternative, fusing position-related values (i.e., position encodings) into neuron outputs. PANs couple themselves to their positions and minimize the possibility of dislocation, even updating on heterogeneous data. We turn on/off PANs to disable/enable the permutation invariance property of neural networks. PANs are tightly coupled with positions when applied to FL, making parameters across clients pre-aligned and facilitating coordinate-based parameter averaging. PANs are algorithm-agnostic and could universally improve existing FL algorithms. Furthermore, “FL with PANs” is simple to implement and computationally friendly.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

13 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Ensemble Distillation for Robust Model Fusion in Federated Learning cites Federated Learning with Matched Averaging
  • Ensemble Distillation for Robust Model Fusion in Federated Learning cites The Non-IID Data Quagmire of Decentralized Machine Learning
  • FedCDA: Federated Learning with Cross-rounds Divergence-aware Aggregation cites Ensemble Distillation for Robust Model Fusion in Federated Learning
  • FedCDA: Federated Learning with Cross-rounds Divergence-aware Aggregation cites Federated Learning with Matched Averaging
  • Federated Learning with Manifold Regularization and Normalized Update Reaggregation cites Federated Learning with Matched Averaging
  • Federated Learning with Manifold Regularization and Normalized Update Reaggregation cites SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
  • Robust Heterogeneous Federated Learning under Data Corruption cites Ensemble Distillation for Robust Model Fusion in Federated Learning
  • Robust Heterogeneous Federated Learning under Data Corruption cites Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach
  • Personalized Federated Learning via Variational Bayesian Inference cites Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning Approach
  • Personalized Federated Learning via Variational Bayesian Inference cites SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
  • The Non-IID Data Quagmire of Decentralized Machine Learning cites Federated Learning with Matched Averaging
  • FedNLR: Federated Learning with Neuron-wise Learning Rates cites SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers13

  1. Personalized Federated Learning via Variational Bayesian InferenceXu Zhang, Yinchuan Li, Wenpeng Li, Kaiyang Guo et al.ICML 2022 · 132 citations
  2. FedCDA: Federated Learning with Cross-rounds Divergence-aware AggregationHaozhao Wang, Haoran Xu, Yichen Li, Yuan Xu et al.ICLR 2024 · 62 citations
  3. Robust Heterogeneous Federated Learning under Data CorruptionXiuwen Fang, Mang Ye, Xiyuan YangICCV 2023 · 44 citations
  4. Federated Learning with Manifold Regularization and Normalized Update ReaggregationXuming An, Li Shen, Han Hu, Yong LuoNeurIPS 2023 · 23 citations
  5. FedNLR: Federated Learning with Neuron-wise Learning RatesHaozhao Wang, Peirong Zheng, Xingshuo Han, Wenchao Xu et al.KDD 2024 · 15 citations

Builds on13

  1. SCAFFOLD: Stochastic Controlled Averaging for Federated LearningSai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi et al.ICML 2020 · 3,875 citations
  2. Ensemble Distillation for Robust Model Fusion in Federated LearningTao Lin, Lingjing Kong, Sebastian U. Stich, Martin JaggiNeurIPS 2020 · 1,615 citations
  3. Federated Learning with Matched AveragingHongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos et al.ICLR 2020 · 1,368 citations
  4. Personalized Federated Learning with Theoretical Guarantees: A Model-Agnostic Meta-Learning ApproachAlireza Fallah, Aryan Mokhtari, Asuman E. OzdaglarNeurIPS 2020 · 1,354 citations
  5. The Non-IID Data Quagmire of Decentralized Machine LearningKevin Hsieh, Amar Phanishayee, Onur Mutlu, Phillip B. GibbonsICML 2020 · 672 citations

Related papers

  1. FedAvg Converges to Zero Training Loss Linearly for Overparameterized Multi-Layer Neural NetworksBingqing Song, Prashant Khanduri, Xinwei Zhang, Jinfeng Yi et al.ICML 2023 · 10 citations
  2. Fed2: Feature-Aligned Federated LearningFuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu et al.KDD 2021 · 57 citations
  3. FedAS: Bridging Inconsistency in Personalized Federated LearningXiyuan Yang, Wenke Huang, Mang YeCVPR 2024 · 69 citations
  4. Class-Wise Federated Averaging for Efficient PersonalizationGyuejeong Lee, Daeyoung ChoiICCV 2025 · 3 citations
  5. Federated Learning Based on Dynamic RegularizationDurmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina et al.ICLR 2021 · 114 citations
Dusk over the sea between two cliffs drawn in fine vertical lines