SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning

Despite Low-Rank Adaptation (LoRA)'s popularity for fine-tuning large models, it often exhibits a noticeable performance gap compared to full fine-tuning, particularly in complex tasks such as mathematical reasoning and code generation. We propose SeedLoRA, a novel fusion approach that bridges this gap by leveraging complementary strengths of multiple LoRA models trained with different random seeds on the same task. Unlike existing model merging methods that focus on combining knowledge from different tasks, SeedLoRA introduces a two-stage fusion strategy specifically designed for single-task scenarios: first identifying and preserving strong shared patterns across models, then performing principled subspace fusion in a unified representation space. Comprehensive experiments on LLaMA2-7B and Mistral-7B demonstrate that SeedLoRA significantly improves performance over individual LoRA models by 4.9% on GSM8K and 6.6% on HumanEval, effectively matching or exceeding full fine-tuning performance while maintaining the efficiency benefits of LoRA. Our analysis reveals that this improvement stems from Seed-LoRA's ability to effectively combine complementary strengths learned by different seeds in a common representation space.