Federated Adaptive Fine-Tuning of Large Language Models with Heterogeneous Quantization and LoRA

Federated learning (FL) with parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), offers a privacy-preserving solution for fine-tuning large language models (LLMs) on edge devices. However, due to the enormous size of LLMs, federated fine-tuning with LoRA still faces significant challenges, including high training latency and substantial memory requirements. To address these challenges, we propose FAH-QLoRA, a novel time- and memory-efficient federated adaptive fine-tuning framework for LLMs, which leverages heterogeneous model quantization and LoRA. The key idea behind FAH-QLoRA is to quantize the base model within LoRA and dynamically adjust the LoRA ranks, enabling efficient fine-tuning on resource-constrained and heterogeneous devices while reducing training latency and maintaining performance. FAH-QLoRA integrates two key techniques: i) dynamically adjusting LoRA ranks across training rounds to promote faster convergence and lower resource usage, and ii) assigning heterogeneous LoRA ranks across devices to mitigate the straggler effect during the FL process, ensuring that heterogeneous resource constraints are met. We analyze the convergence of FAH-QLoRA under general non-convex and non-IID FL settings. Extensive experiments demonstrate that FAH-QLoRA can reduce training time by up to 45.86% and memory usage by up to 44.15% compared to baseline methods.