Less Is More in Federated Continual Learning: RieSelect for Conflict-Aware Layer Selection in LLMs
Wenqi Qiu, Yipeng Zhou, Lin Zhu, Laizhong Cui
摘要
Federated continual learning (FCL) of large language models on edge devices is constrained by a communication--stability--plasticity trilemma. We reveal a less-is-more phenomenon: beyond a moderate layer upload ratio, stability loss offsets saturated plasticity gains, so overall continual performance no longer improves. Moreover, layer-wise conflict is heavy-tailed and concentrates in a few layers; denser uplink increasingly includes these layers, which disproportionately drives forgetting and motivates selective sparse communication. Therefore, we introduce RieSelect, which treats stability as staying within a Fisher-metric safe basin around historical solutions. Under this safe-basin constraint, we derive a layer-wise conflict score and a closed-form certified safe step size for finite local updates, and formulate selective uplink as a knapsack-based utility--risk selection, balancing plasticity gains against stability risks. Extensive experiments show that, under a per-round uplink budget, RieSelect achieves the best performance across task orders. Beyond this matched-budget setting, under standard communication protocols, RieSelect improves average accuracy by 18.99–28.14 points while reducing total uplink by 53–115.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper16
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu 等ICLR 2022 · 被引用 18,833 次
- FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 被引用 2,600 次
- Federated Continual Learning with Weighted Inter-client TransferJaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang 等ICML 2021 · 被引用 303 次
- Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP TasksYizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi 等EMNLP 2022 · 被引用 238 次
- Federated Class-Incremental LearningJiahua Dong, Lixu Wang, Zhen Fang, Gan Sun 等CVPR 2022 · 被引用 197 次
相关 Paper
- Budgeted Online Continual Learning by Adaptive Layer Freezing and Frequency-based SamplingMinhyuk Seo, Hyunseo Koh, Jonghyun ChoiICLR 2025
- Resource-Constrained Federated Continual Learning: What Does Matter?Yichen Li, Yuying Wang, Jiahua Dong, Haozhao Wang 等NeurIPS 2025 · 被引用 7 次
- SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language ModelsJinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao 等EMNLP 2024 · 被引用 4 次
- ElasticFed: Collaborative Large-Small Transformer Training for Federated Continual Learning at EdgeYunlai Cheng, Rui Han, Qinglong Zhang, Chi Harold LiuINFOCOM 2025
- Federated Continual Learning via Orchestrating Multi-Scale ExpertiseXiaoyang Yi, Yang Liu, Binhan Yang, Jian Jun ZhangNeurIPS 2025
