Lune

EMNLP Findings2025

Riemannian Optimization for LoRA on the Stiefel Manifold

JuneYoung Park, Minjae Kang, Seongbae Lee, Haegang Lee, Seongwan Kim, Jaeho Lee

Abstract

While powerful, large language models (LLMs) present significant fine-tuning challenges due to their size. Parameter-efficient fine-tuning (PEFT) methods like LoRA provide solutions, yet suffer from critical optimizer inefficiencies; notably basis redundancy in LoRA's B matrix when using AdamW, which fundamentally limits performance. We address this by optimizing the B matrix on the Stiefel manifold, imposing explicit orthogonality constraints that achieve near-perfect orthogonality and full effective rank. This geometric approach dramatically enhances parameter efficiency and representational capacity. Our Stiefel optimizer consistently outperforms AdamW across benchmarks with both LoRA and DoRA, demonstrating that geometric constraints are the key to unlocking LoRA's full potential for effective LLM finetuning.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning cites Language Models are Few-Shot Learners
  • Balanced LoRA: Removing Parameter Invariance to Accelerate Convergence cites Language Models are Few-Shot Learners
  • VeRA: Vector-based Random Matrix Adaptation cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Riemannian stochastic optimization for sufficient dimension reductionThibault Pautrel, François PortierICML 2026
  2. Balanced LoRA: Removing Parameter Invariance to Accelerate ConvergenceValérie Castin, Kimia Nadjahi, Pierre Ablin, Gabriel PeyréICML 2026

Builds on11

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. DoRA: Weight-Decomposed Low-Rank AdaptationShih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov et al.ICML 2024 · 820 citations
  3. VeRA: Vector-based Random Matrix AdaptationDawid Jan Kopiczko, Tijmen Blankevoort, Yuki M. AsanoICLR 2024 · 308 citations
  4. HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-TuningChunlin Tian, Zhan Shi, Zhijiang Guo, Li Li et al.NeurIPS 2024 · 172 citations
  5. LoRA vs Full Fine-tuning: An Illusion of EquivalenceReece Shuttleworth, Jacob Andreas, Antonio Torralba, Pratyusha SharmaNeurIPS 2025 · 152 citations

Related papers

  1. StelLA: Subspace Learning in Low-rank Adaptation using Stiefel ManifoldZhizhong Li, Sina Sajadmanesh, Jingtao Li, Lingjuan LyuNeurIPS 2025 · 16 citations
  2. Sparsity May Be All You Need: Sparse Random Parameter AdaptationJesus Rios, Pierre L. Dognin, Ronny Luss, Karthikeyan Natesan RamamurthyEMNLP Findings 2025
  3. Riemannian Preconditioned LoRA for Fine-Tuning Foundation ModelsFangzhao Zhang, Mert PilanciICML 2024 · 43 citations
  4. UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter Efficient Fine-Tuning of Large ModelsXueyan Zhang, Jinman Zhao, Zhifei Yang, Yibo Zhong et al.ACL 2025 · 15 citations
  5. HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language ModelsQiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines