Lune

NeurIPS2024Top-tier venue

Heterogeneity-Guided Client Sampling: Towards Fast and Efficient Non-IID Federated Learning

Huancheng Chen, Haris Vikalo

Abstract

Statistical heterogeneity of data present at client devices in a federated learning (FL) system renders the training of a global model in such systems difficult. Particularly challenging are the settings where due to communication resource constraints only a small fraction of clients can participate in any given round of FL. Recent approaches to training a global model in FL systems with non-IID data have focused on developing client selection methods that aim to sample clients with more informative updates of the model. However, existing client selection techniques either introduce significant computation overhead or perform well only in the scenarios where clients have data with similar heterogeneity profiles. In this paper, we propose HiCS-FL (Federated Learning via Hierarchical Clustered Sampling), a novel client selection method in which the server estimates statistical heterogeneity of a client's data using the client's update of the network's output layer and relies on this information to cluster and sample the clients. We analyze the ability of the proposed techniques to compare heterogeneity of different datasets, and characterize convergence of the training process that deploys the introduced client selection method. Extensive experimental results demonstrate that in non-IID settings HiCS-FL achieves faster convergence than state-of-the-art FL client selection schemes. Notably, HiCS-FL drastically reduces computation cost compared to existing selection schemes and is adaptable to different heterogeneity scenarios.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization cites Federated Learning with Matched Averaging
  • Domain Sensitive Federated Learning with Fisher-Informed Pruning cites SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
  • Ditto: Fair and Robust Federated Learning Through Personalization cites SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
  • Exploiting Shared Representations for Personalized Federated Learning cites SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
  • Exploiting Shared Representations for Personalized Federated Learning cites Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
  • Exploiting Shared Representations for Personalized Federated Learning cites Ditto: Fair and Robust Federated Learning Through Personalization
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Mind the Gap: Confidence Discrepancy Can Guide Federated Semi-Supervised Learning Across Pseudo-MismatchYijie Liu, Xinyi Shang, Yiqun Zhang, Yang Lu et al.CVPR 2025
  2. Domain Sensitive Federated Learning with Fisher-Informed PruningChenchen Lin, Wenhao Yuan, Zhengji Xu, Xuehe WangCVPR 2026

Builds on14

  1. SCAFFOLD: Stochastic Controlled Averaging for Federated LearningSai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi et al.ICML 2020 · 3,875 citations
  2. Tackling the Objective Inconsistency Problem in Heterogeneous Federated OptimizationJianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi et al.NeurIPS 2020 · 2,231 citations
  3. Federated Learning with Matched AveragingHongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos et al.ICLR 2020 · 1,368 citations
  4. Ditto: Fair and Robust Federated Learning Through PersonalizationTian Li, Shengyuan Hu, Ahmad Beirami, Virginia SmithICML 2021 · 1,313 citations
  5. Exploiting Shared Representations for Personalized Federated LearningLiam Collins, Hamed Hassani, Aryan Mokhtari, Sanjay ShakkottaiICML 2021 · 1,081 citations

Related papers

  1. Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated LearningYann Fraboni, Richard Vidal, Laetitia Kameni, Marco LorenziICML 2021 · 249 citations
  2. Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client SamplingBing Luo, Wenli Xiao, Shiqiang Wang, Jianwei Huang et al.INFOCOM 2022 · 224 citations
  3. Heterogeneity-Aware Federated Learning with Adaptive Client Selection and Gradient CompressionZhida Jiang, Yang Xu, Hongli Xu, Zhiyuan Wang et al.INFOCOM 2023 · 43 citations
  4. TiFL: A Tier-based Federated Learning SystemZheng Chai, Ahsan Ali, Syed Zawad, Stacey Truex et al.HPDC 2020 · 330 citations
  5. Structured Federated Learning through Clustered Additive ModelingJie Ma, Tianyi Zhou, Guodong Long, Jing Jiang et al.NeurIPS 2023 · 28 citations
Dusk over the sea between two cliffs drawn in fine vertical lines