TransKV: A Data-Driven Pruning Method for Large Foundation Models

Large Foundation Models (LFMs) have achieved remarkable success across various domains, yet their enormous parameter counts severely hinder deployment on resourceconstrained devices. Recent pruning methods leverage lowrank structures in attention mechanisms, with SOTA techniques like CLOVER and Olica targeting redundancies in weight pairs (e.g., W i Q (W i K ) ⊤ from i-th attention head) to enable more aggressive compression than single-matrix pruning. However, these static approaches overlook datadriven interactions, as singular value distributions of static weight pairs differ significantly from data-driven counterparts (e.g., XW i Q (W i K ) ⊤ X ⊤ ), often yielding suboptimal performance. To address this, we introduce TransKV, a data-driven pruning framework that derives a projection matrix from keys (XW i K ) and values (XW i V ), seamlessly integrating it into weight-pair computations for precise low-rank truncation. TransKV natively supports core LFM components, including MHA, GQA, RMSNorm, and RoPE, ensuring compatibility with contemporary LFMs. Extensive experiments on diverse benchmarks across multiple domains validate TransKV's superiority, outperforming SOTA methods on various LFMs at equivalent compression ratios. The code is available on GitHub through: