On the Representation Geometry of LoRA Model Merging

Low-Rank Adaptation (LoRA) is widely used for parameter-efficient fine-tuning, yet merging multiple task-specific LoRA updates without additional training remains challenging. Most existing LoRA merging methods rely on SVDbased alignment, which emphasizes globally shared structure across tasks. In this work, we show that LoRA merging performance can be further improved by combining SVD with CUR decomposition. Through a representationlevel analysis, we find that SVD-based decompositions primarily model shared components across tasks, while CUR-based decompositions better preserve task-specific and localized updates. These two perspectives are geometrically misaligned and exhibit complementary advantages, revealing an inherent trade-off between capturing shared structure and preserving task-specific information in LoRA model merging. Guided by this analysis, we propose a training-free merging procedure that explicitly combines the shared structure captured by SVD with the task-specific components preserved by CUR. Experiments on both vision and language benchmarks demonstrate consistent improvements over existing gradient-free LoRA merging methods. Our code is available at https: //github.com/lcytoronto/knots_cur .