DisLoRA: Task-specific Low-Rank Adaptation via Orthogonal Basis from Singular Value Decomposition

Parameter-efficient fine-tuning (PEFT) of large language models (LLMs) is critical for adapting to diverse downstream tasks with minimal computational cost. We propose Directional-SVD Low-Rank Adaptation (DisLoRA), a novel PEFT framework that leverages Singular Value Decomposition (SVD) to decompose pretrained weight matrices into orthogonal backbone and task-specific subspaces, enabling precise capture of task-specific directions (TSDs). By dynamically identifying TSDs and employing adaptive soft orthogonal regularization with mean-normalization mechanism, DisLoRA balances task-specific and orthogonal losses without manual tuning, ensuring robust training stability. Extensive experiments on GLUE and Commonsense Reasoning benchmarks demonstrate that Dis-LoRA surpasses established PEFT methods, including LoRA, PiSSA, DoRA, LoRA-Dash, SORSA and MiLoRA. DisLoRA achieves superior performance on multiple individual GLUE datasets, surpassing baselines by up to 10.28% on SST-2 and 3.28% on CoLA, and consistently attains higher average accuracy than baselines across Commonsense Reasoning Tasks, with a maximum gain of 3.1%. Additionally, We also evaluated DisLoRA on the AQUA-RAT mathematical dataset, where it achieved the highest accuracy across all tested ranks. These results demonstrate DisLoRA's performance in efficient and high-performing LLM adaptation for domain-specific tasks while preserving generalization.