EMNLP Findings2025
Optimizing Cross-Client Domain Coverage for Federated Instruction Tuning of Large Language Models
Zezhou Wang, Yaxin Du, Xingjun Ma, Yu-Gang Jiang, Zhuzhong Qian, Siheng Chen
Abstract
Federated domain-specific instruction tuning (FedDIT) for large language models (LLMs) aims to enhance performance in specialized domains using distributed private and limited data, yet identifying key performance drivers and optimal augmentation strategies remains challenging. We empirically establish that cross-client domain coverage, rather than data heterogeneity, is the pivotal factor. We then introduce FedDCA, an algorithm that explicitly maximizes this coverage through diversity-oriented client center selection and retrieval-based augmentation, constructing diverse, non-redundant cross-client instruction sets. Extensive experiments across multiple domains demonstrate FedDCA's superiority over eleven baselines, achieving performance gains of up to 29.19% and domain coverage improvements of 4.82%-21.36%. FedDCA maintains its effectiveness in diverse and challenging scenarios, including data selection, held-out settings where task-specific public data is scarce and various data heterogeneity, with manageable privacy risks. This work clarifies critical FedDIT dynamics and presents FedDCA as an effective, privacy-preserving, and scalable solution for advancing domain-specific LLM tuning.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on16
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
- SCAFFOLD: Stochastic Controlled Averaging for Federated LearningSai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank J. Reddi et al.ICML 2020 · 3,875 citations
- On the Convergence of FedAvg on Non-IID DataXiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang et al.ICLR 2020 · 2,930 citations
- Extracting Training Data from Large Language ModelsNicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski et al.USENIX Security 2021 · 2,866 citations
Related papers
- Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active ExplorationFanqi Wan, Xinting Huang, Tao Yang, Xiaojun Quan et al.EMNLP 2023 · 2 citations
- Federated Data-Efficient Instruction Tuning for Large Language ModelsZhen Qin, Zhaomin Wu, Bingsheng He, Shuiguang DengACL Findings 2025 · 5 citations
- Data Diversity Matters for Robust Instruction TuningAlexander Bukharin, Shiyang Li, Zhengyang Wang, Jingfeng Yang et al.EMNLP Findings 2024 · 13 citations
- Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model ArchitecturesYicheng Zhang, Zhen Qin, Zhaomin Wu, Jian Hou et al.WWW 2026 · 9 citations
- Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language ModelsYangxu Liao, Wenke Huang, Guancheng Wan, Jian Liang et al.ICML 2025
