Lune

KDD2021Top-tier venue

Fed2: Feature-Aligned Federated Learning

Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, Xiang Chen

Abstract

Federated learning learns from scattered data by fusing collaborative models from local nodes. However, conventional coordinatebased model averaging by FedAvg ignored the random information encoded per parameter and may suffer from structural feature misalignment. In this work, we propose 𝐹𝑒𝑑 2 , a feature-aligned federated learning framework to resolve this issue by establishing a firm structure-feature alignment across the collaborative models. 𝐹𝑒𝑑 2 is composed of two major designs: First, we design a feature-oriented model structure adaptation method to ensure explicit feature allocation in different neural network structures. Applying the structure adaptation to collaborative models, matchable structures with similar feature information can be initialized at the very early training stage. During the federated learning process, we then propose a feature paired averaging scheme to guarantee aligned feature distribution and maintain no feature fusion conflicts under either IID or non-IID scenarios. Eventually, 𝐹𝑒𝑑 2 could effectively enhance the federated learning convergence performance under extensive homoand heterogeneous settings, providing excellent convergence speed, accuracy, and computation/communication efficiency.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

19 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Towards Model Agnostic Federated Learning Using Knowledge Distillation cites Model Fusion via Optimal Transport
  • FedCDA: Federated Learning with Cross-rounds Divergence-aware Aggregation cites Federated Learning with Matched Averaging
  • FedCDA: Federated Learning with Cross-rounds Divergence-aware Aggregation cites Model Fusion via Optimal Transport
  • Federated Learning with Matched Averaging cites Model Fusion via Optimal Transport
  • Model Fusion via Optimal Transport cites Federated Learning with Matched Averaging
  • Preservation of the Global Knowledge by Not-True Distillation in Federated Learning cites Federated Learning with Matched Averaging
  • Preservation of the Global Knowledge by Not-True Distillation in Federated Learning cites On the Convergence of FedAvg on Non-IID Data
  • Federated Model Heterogeneous Matryoshka Representation Learning cites FedGH: Heterogeneous Federated Learning with Generalized Global Header
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers19

  1. Preservation of the Global Knowledge by Not-True Distillation in Federated LearningGihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae et al.NeurIPS 2022 · 235 citations
  2. FedGH: Heterogeneous Federated Learning with Generalized Global HeaderLiping Yi, Gang Wang, Xiaoguang Liu, Zhuan Shi et al.ACM MM 2023 · 137 citations
  3. FedCDA: Federated Learning with Cross-rounds Divergence-aware AggregationHaozhao Wang, Haoran Xu, Yichen Li, Yuan Xu et al.ICLR 2024 · 62 citations
  4. Towards Model Agnostic Federated Learning Using Knowledge DistillationAndrei Afonin, Sai Praneeth KarimireddyICLR 2022 · 55 citations
  5. Federated Model Heterogeneous Matryoshka Representation LearningLiping Yi, Han Yu, Chao Ren, Gang Wang et al.NeurIPS 2024 · 46 citations

Builds on3

  1. On the Convergence of FedAvg on Non-IID DataXiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang et al.ICLR 2020 · 2,930 citations
  2. Federated Learning with Matched AveragingHongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos et al.ICLR 2020 · 1,368 citations
  3. Model Fusion via Optimal TransportSidak Pal Singh, Martin JaggiNeurIPS 2020 · 330 citations

Related papers

  1. Distribution-Regularized Federated Learning on Non-IID DataYansheng Wang, Yongxin Tong, Zimu Zhou, Ruisheng Zhang et al.ICDE 2023 · 31 citations
  2. Understanding Convergence and Generalization in Federated Learning through Feature Learning TheoryWei Huang, Ye Shi, Zhongyi Cai, Taiji SuzukiICLR 2024 · 17 citations
  3. Federated Learning with Position-Aware NeuronsXin-Chun Li, Yichu Xu, Shaoming Song, Bingshuai Li et al.CVPR 2022 · 23 citations
  4. On the Effectiveness of Partial Variance Reduction in Federated Learning with Heterogeneous DataBo Li, Mikkel N. Schmidt, Tommy S. Alstrøm, Sebastian U. StichCVPR 2023
  5. Federated Learning for Non-IID Data via Unified Feature Learning and Optimization Objective AlignmentLin Zhang, Yong Luo, Yan Bai, Bo Du et al.ICCV 2021 · 98 citations
Dusk over the sea between two cliffs drawn in fine vertical lines