Bayesian-LoRA: Probabilistic Low-Rank Adaptation of Large Language Models

Large language models are typically optimized for accuracy and, therefore, will guess even when uncertain about their predictions. This problem becomes especially pronounced when the model is fine-tuned on small datasets, which often causes overfitting and results in a tendency toward miscalibration. In this work, we introduce Bayesian-LoRA, which reformulates the deterministic LoRA update as a probabilistic low-rank representation inspired by Sparse Gaussian Processes (SGP). We identify a structural isomorphism between LoRA's factorization and Kronecker-factored SGP posteriors, and show that LoRA emerges as a limiting case when posterior uncertainty collapses. We conduct extensive experiments on various LLM architectures across commonsense reasoning, language modeling, and mathematical reasoning benchmarks. With only approximately 0.42M additional parameters and ${\approx}1.2{\times}$ training cost relative to standard LoRA, Bayesian-LoRA significantly improves calibration across models from 7B up to 30B, achieving up to 84% Expected Calibration Error (ECE) and 76% Negative Log-Likelihood (NLL) reduction while maintaining competitive accuracy for both in-distribution and out-of-distribution (OoD) evaluations.