GeoLoRA: Geometric integration for parameter efficient fine-tuning

Large language models and text encoders increasingly power webbased information systems in the social sciences, including digital libraries, data catalogues, and search interfaces used by researchers, policymakers, and civil society. Full fine-tuning is often computationally and energy intensive, which can be prohibitive for smaller institutions and non-profit organizations in the Web4Good ecosystem. Parameter-Efficient Fine-Tuning (PEFT), especially Low-Rank Adaptation (LoRA), reduces this cost by updating only a small number of parameters. We show that the standard LoRA update Δ𝑊 = 𝐵𝐴 ⊤ has geometric drawbacks: gauge freedom, scale ambiguity, and a tendency toward rank collapse. We introduce Or-thoGeoLoRA, which enforces an SVD-like form Δ𝑊 = 𝐵Σ𝐴 ⊤ by constraining the low-rank factors to be orthogonal (Stiefel manifold). A geometric reparameterization implements this constraint while remaining compatible with standard optimizers such as Adam and existing fine-tuning pipelines. We also propose a benchmark for hierarchical concept retrieval over the European Language Social Science Thesaurus (ELSST), widely used to organize social science resources in digital repositories. Experiments with a multilingual sentence encoder show that OrthoGeoLoRA outperforms standard LoRA and several strong PEFT variants on ranking metrics under the same low-rank budget, offering a more compute-and parameterefficient path to adapt foundation models in resource-constrained settings.