The Geometry of Narrow Fine-Tuning Degradation: Trajectory Lock-in and Spectral Bifurcation
Jia Liu, Jiaxin Luo, Xinhao Qiu, Yixue Hao, Min Chen
摘要
Magnitude-based stability proxies such as parameter drift are widely used in narrow-task fine-tuning, yet they do not reliably indicate degradation of broad capabilities. We identify trajectory lock-in: under fixed training conditions for narrow adaptation, the joint evolution of task loss and broad generalization collapses onto a shared low-dimensional degradation curve, so many stabilizers primarily change the rate of progress along this curve rather than altering the curve itself. This yields a drift paradox, in which comparable Euclidean displacement can still correspond to divergent generalization outcomes. To diagnose the underlying structure, we introduce objective-agnostic geometric probes that track the effective update subspace, together with an online harm signal that reflects curvature-dominated channeling toward directions associated with broad degradation. Finally, we show that escaping lock-in requires a spectral bifurcation, namely a qualitative reorientation of the update subspace toward softer curvature modes, thereby improving broad generalization while maintaining matched training performance. We validate these findings across model scales and modalities in narrow-task settings, and report practical deployment procedures and overhead measurements.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper14
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu 等ICLR 2022 · 被引用 18,833 次
- Sharpness-aware Minimization for Efficiently Improving GeneralizationPierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam NeyshaburICLR 2021 · 被引用 1,861 次
- Don't Stop Pretraining: Adapt Language Models to Domains and TasksSuchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo 等ACL 2020 · 被引用 93 次
- FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-ExpertsHeming Zou, Yunliang Zang, Wutong Xu, Yao Zhu 等NeurIPS 2025 · 被引用 38 次
- Adaptive Budget Allocation for Parameter-Efficient Fine-TuningQingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He 等ICLR 2023 · 被引用 32 次
相关 Paper
- Blocking the Leakage: Manifold-Aware Gradient Projection for Long-Horizon Test-Time AdaptationHaoyu Xiong, Chengchao Wang, ZhongQiang Wang, Huang He 等ICML 2026
- GradientStabilizer: Fix the Norm, Not the GradientTianjin Huang, Zhangyang “Atlas” Wang, Haotian Hu, Zhenyu Zhang 等ICML 2026
- Geometry-Aware Update Control for Robust OptimizationYitong Ma, Mandi Li, Gang YanKDD 2026
- Escaping the Subspace Trap: The Role of Optimizer Geometry in Model Width ExpansionJiabei Chen, Haoyu Wang, Yang Yu, Yao Xu 等ICML 2026
- Stabilizing Feature Geometry in Noisy Pretrained Models for Robust Downstream TasksQuanyu Zhang, Zhongyi Han, Hao Sun, Yongshun Gong 等CVPR 2026
