LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization

Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLMs that reduces memory requirements. However, current LoRA optimizers lack transformation invariance, which leads to weight updates that depend on how the two LoRA factors are scaled or rotated. This deficiency leads to inefficient learning and sub-optimal solutions in practice. This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization, which achieves transformation invariance while being computationally efficient. We provide theoretical analysis to demonstrate the benefit of our method and conduct experiments on various LLM tasks with different models including Gemma 2B, 7B, and mT5-XXL. The results demonstrate consistent improvements over existing optimizers. For example, replacing Adam with LoRA-RITE during LoRA fine-tuning of Gemma-2B yields 4.6% accuracy gain on Super-Natural Instructions and 3.5% accuracy gain across four other LLM benchmarks (HellaSwag, ArcChallenge, GSM8K, OpenBookQA).