Fed2: Feature-Aligned Federated Learning
Fuxun Yu, Weishan Zhang, Zhuwei Qin, Zirui Xu, Di Wang, Chenchen Liu, Zhi Tian, Xiang Chen
摘要
Federated learning learns from scattered data by fusing collaborative models from local nodes. However, conventional coordinatebased model averaging by FedAvg ignored the random information encoded per parameter and may suffer from structural feature misalignment. In this work, we propose 𝐹𝑒𝑑 2 , a feature-aligned federated learning framework to resolve this issue by establishing a firm structure-feature alignment across the collaborative models. 𝐹𝑒𝑑 2 is composed of two major designs: First, we design a feature-oriented model structure adaptation method to ensure explicit feature allocation in different neural network structures. Applying the structure adaptation to collaborative models, matchable structures with similar feature information can be initialized at the very early training stage. During the federated learning process, we then propose a feature paired averaging scheme to guarantee aligned feature distribution and maintain no feature fusion conflicts under either IID or non-IID scenarios. Eventually, 𝐹𝑒𝑑 2 could effectively enhance the federated learning convergence performance under extensive homoand heterogeneous settings, providing excellent convergence speed, accuracy, and computation/communication efficiency.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 19 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- Towards Model Agnostic Federated Learning Using Knowledge Distillation 引用了 Model Fusion via Optimal Transport
- FedCDA: Federated Learning with Cross-rounds Divergence-aware Aggregation 引用了 Federated Learning with Matched Averaging
- FedCDA: Federated Learning with Cross-rounds Divergence-aware Aggregation 引用了 Model Fusion via Optimal Transport
- Federated Learning with Matched Averaging 引用了 Model Fusion via Optimal Transport
- Model Fusion via Optimal Transport 引用了 Federated Learning with Matched Averaging
- Preservation of the Global Knowledge by Not-True Distillation in Federated Learning 引用了 Federated Learning with Matched Averaging
- Preservation of the Global Knowledge by Not-True Distillation in Federated Learning 引用了 On the Convergence of FedAvg on Non-IID Data
- Federated Model Heterogeneous Matryoshka Representation Learning 引用了 FedGH: Heterogeneous Federated Learning with Generalized Global Header
引用它的顶会 Paper19
- Preservation of the Global Knowledge by Not-True Distillation in Federated LearningGihun Lee, Minchan Jeong, Yongjin Shin, Sangmin Bae 等NeurIPS 2022 · 235 次引用
- FedGH: Heterogeneous Federated Learning with Generalized Global HeaderLiping Yi, Gang Wang, Xiaoguang Liu, Zhuan Shi 等ACM MM 2023 · 137 次引用
- FedCDA: Federated Learning with Cross-rounds Divergence-aware AggregationHaozhao Wang, Haoran Xu, Yichen Li, Yuan Xu 等ICLR 2024 · 62 次引用
- Towards Model Agnostic Federated Learning Using Knowledge DistillationAndrei Afonin, Sai Praneeth KarimireddyICLR 2022 · 55 次引用
- Federated Model Heterogeneous Matryoshka Representation LearningLiping Yi, Han Yu, Chao Ren, Gang Wang 等NeurIPS 2024 · 46 次引用
它引用的 Paper3
- On the Convergence of FedAvg on Non-IID DataXiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang 等ICLR 2020 · 2,930 次引用
- Federated Learning with Matched AveragingHongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos 等ICLR 2020 · 1,368 次引用
- Model Fusion via Optimal TransportSidak Pal Singh, Martin JaggiNeurIPS 2020 · 330 次引用
相关 Paper
- Distribution-Regularized Federated Learning on Non-IID DataYansheng Wang, Yongxin Tong, Zimu Zhou, Ruisheng Zhang 等ICDE 2023 · 31 次引用
- Understanding Convergence and Generalization in Federated Learning through Feature Learning TheoryWei Huang, Ye Shi, Zhongyi Cai, Taiji SuzukiICLR 2024 · 17 次引用
- Federated Learning with Position-Aware NeuronsXin-Chun Li, Yichu Xu, Shaoming Song, Bingshuai Li 等CVPR 2022 · 23 次引用
- On the Effectiveness of Partial Variance Reduction in Federated Learning with Heterogeneous DataBo Li, Mikkel N. Schmidt, Tommy S. Alstrøm, Sebastian U. StichCVPR 2023
- Federated Learning for Non-IID Data via Unified Feature Learning and Optimization Objective AlignmentLin Zhang, Yong Luo, Yan Bai, Bo Du 等ICCV 2021 · 98 次引用
