The Geometry of Narrow Fine-Tuning Degradation: Trajectory Lock-in and Spectral Bifurcation
Jia Liu, Jiaxin Luo, Xinhao Qiu, Yixue Hao, Min Chen
Abstract
Magnitude-based stability proxies such as parameter drift are widely used in narrow-task fine-tuning, yet they do not reliably indicate degradation of broad capabilities. We identify trajectory lock-in: under fixed training conditions for narrow adaptation, the joint evolution of task loss and broad generalization collapses onto a shared low-dimensional degradation curve, so many stabilizers primarily change the rate of progress along this curve rather than altering the curve itself. This yields a drift paradox, in which comparable Euclidean displacement can still correspond to divergent generalization outcomes. To diagnose the underlying structure, we introduce objective-agnostic geometric probes that track the effective update subspace, together with an online harm signal that reflects curvature-dominated channeling toward directions associated with broad degradation. Finally, we show that escaping lock-in requires a spectral bifurcation, namely a qualitative reorientation of the update subspace toward softer curvature modes, thereby improving broad generalization while maintaining matched training performance. We validate these findings across model scales and modalities in narrow-task settings, and report practical deployment procedures and overhead measurements.
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 0fc8b4cd-5e15-4638-957b-887163587341Builds on14
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Sharpness-aware Minimization for Efficiently Improving GeneralizationPierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam NeyshaburICLR 2021 · 1,861 citations
- Don't Stop Pretraining: Adapt Language Models to Domains and TasksSuchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo et al.ACL 2020 · 93 citations
- FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-ExpertsHeming Zou, Yunliang Zang, Wutong Xu, Yao Zhu et al.NeurIPS 2025 · 38 citations
- Adaptive Budget Allocation for Parameter-Efficient Fine-TuningQingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He et al.ICLR 2023 · 32 citations
Related papers
- Blocking the Leakage: Manifold-Aware Gradient Projection for Long-Horizon Test-Time AdaptationHaoyu Xiong, Chengchao Wang, ZhongQiang Wang, Huang He et al.ICML 2026
- GradientStabilizer: Fix the Norm, Not the GradientTianjin Huang, Zhangyang “Atlas” Wang, Haotian Hu, Zhenyu Zhang et al.ICML 2026
- Geometry-Aware Update Control for Robust OptimizationYitong Ma, Mandi Li, Gang YanKDD 2026
- Escaping the Subspace Trap: The Role of Optimizer Geometry in Model Width ExpansionJiabei Chen, Haoyu Wang, Yang Yu, Yao Xu et al.ICML 2026
- Stabilizing Feature Geometry in Noisy Pretrained Models for Robust Downstream TasksQuanyu Zhang, Zhongyi Han, Hao Sun, Yongshun Gong et al.CVPR 2026
