Less Is More in Federated Continual Learning: RieSelect for Conflict-Aware Layer Selection in LLMs
Wenqi Qiu, Yipeng Zhou, Lin Zhu, Laizhong Cui
Abstract
Federated continual learning (FCL) of large language models on edge devices is constrained by a communication--stability--plasticity trilemma. We reveal a less-is-more phenomenon: beyond a moderate layer upload ratio, stability loss offsets saturated plasticity gains, so overall continual performance no longer improves. Moreover, layer-wise conflict is heavy-tailed and concentrates in a few layers; denser uplink increasingly includes these layers, which disproportionately drives forgetting and motivates selective sparse communication. Therefore, we introduce RieSelect, which treats stability as staying within a Fisher-metric safe basin around historical solutions. Under this safe-basin constraint, we derive a layer-wise conflict score and a closed-form certified safe step size for finite local updates, and formulate selective uplink as a knapsack-based utility--risk selection, balancing plasticity gains against stability risks. Extensive experiments show that, under a per-round uplink budget, RieSelect achieves the best performance across task orders. Beyond this matched-budget setting, under standard communication protocols, RieSelect improves average accuracy by 18.99–28.14 points while reducing total uplink by 53–115.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 3f9a98a2-8b21-4c50-8446-4948eb6a0214Builds on16
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
- Federated Continual Learning with Weighted Inter-client TransferJaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang et al.ICML 2021 · 303 citations
- Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP TasksYizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi et al.EMNLP 2022 · 238 citations
- Federated Class-Incremental LearningJiahua Dong, Lixu Wang, Zhen Fang, Gan Sun et al.CVPR 2022 · 197 citations
Related papers
- Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based SamplingMinhyuk Seo, Hyunseo Koh, Jonghyun ChoiICLR 2025
- Resource-Constrained Federated Continual Learning: What Does Matter?Yichen Li, Yuying Wang, Jiahua Dong, Haozhao Wang et al.NeurIPS 2025 · 7 citations
- SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language ModelsJinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao et al.EMNLP 2024 · 4 citations
- ElasticFed: Collaborative Large-Small Transformer Training for Federated Continual Learning at EdgeYunlai Cheng, Rui Han, Qinglong Zhang, Chi Harold LiuINFOCOM 2025
- Federated Continual Learning via Orchestrating Multi-Scale ExpertiseXiaoyang Yi, Yang Liu, Binhan Yang, Jian Jun ZhangNeurIPS 2025
