Representation Drift Compensation: A Near-Zero Inference Cost Enhancement for LLM Decomposition

While low-rank decomposition offers potential for reducing LLM parameters, maintaining the original capabilities remains a significant challenge. In this work, we identify and formalize a key overlooked issue in LLM decomposition: representation drift. We show that approximation errors introduced by decomposition propagate and amplify non-linearly through the deep layers of the transformer architecture, progressively distorting internal representations and degrading downstream performance. To mitigate this, we introduce a conceptually simple but principled compensation mechanism, named ``Decomper'', that operates by suppressing error at its source. By learning to align the output distribution of decomposed transformer blocks with their original counterparts, our method effectively counteracts representation drift, achieving notable performance recovery with near-zero inference overhead. Extensive experiments in OPT, LLaMA-2/3, and Qwen exhibit remarkable improvements. For instance, on LLaMA-3-8B and OPT-13B at 40% compression, perplexity is reduced by more than 70% while reasoning task accuracy improves by over 10%. Our code is available at this URL.