FedDynMask: Efficient Federated Fine-Tuning for Edge LLMs via Dynamic Sparse Masking

Deploying large language models (LLMs) on edge devices often demands effective fine-tuning to adapt to local requirements, yet confronts two key challenges: data heterogeneity and resource constraints. Existing solutions fall short: traditional federated learning methods, inherently designed for small-scale models, fail to scale to the size and complexity of LLMs. Meanwhile, current federated fine-tuning approaches for LLMs fail to adequately address the stringent resource constraints of edge devices. To address these, we propose FedDynMask——a framework designed specifically for efficient federated fine-tuning of LLMs on edge devices. FedDynMask couples Dynamic Personalized Sparse Masking (DPSM) on edge devices with Masked Intersection Aggregation (MIA) on the server via dynamic mask updates. DPSM enables devices to dynamically identify and retain only task-essential parameters by leveraging local data distributions, reducing computational overhead and enabling local adaptation. MIA further reduces communication overhead by identifying consistent parameters through masked intersection, preventing global model performance degradation caused by heterogeneous data. Compared to baselines, this design reduces communication overhead to 0.366M and 0.522M for Llama3.2-3B and Llama3-8B models, respectively, and computational overhead to 3.67M and 5.24M. It also achieves a 1.36% accuracy improvement over other methods, making federated fine-tuning of LLMs feasible on resource-constrained edge devices.