Context-Conditioned Masked LoRA: Dynamic Rank Routing for Compute-Efficient Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning methods such as LoRA reduce trainable parameters, but still apply dense low-rank updates per token, leaving adaptation compute largely fixed once rank is set. We propose Context-Conditioned Masked LoRA (CCM-LoRA), which learns a lightweight router that activates an inputdependent subset of LoRA rank directions, turning LoRA into dynamic rank routing and enabling contextual sparsity in fine-tuning and inference. CCM-LoRA is trained with a budget-constrained objective that targets an expected effective rank (or FLOPs) while regularizing routing to avoid degenerate alwayson/off masks. Across public NLU and multilingual benchmarks, CCM-LoRA improves the accuracy-efficiency Pareto frontier versus static-rank LoRA and adaptive-rank baselines, matching or improving task performance at lower inference-time effective rank. We also provide a reproducible profiling protocol and analyses of rank usage, router overhead, and robustness under domain and language shift.