PermLLM: Learnable Channel Permutation for N: M Sparse Large Language Models

Channel permutation is a powerful technique for enhancing the accuracy of N:M sparse models by reordering the channels of weight matrices to prioritize the retention of important weights. However, traditional channel permutation methods rely on handcrafted quality metrics, which often fail to accurately capture the true impact of pruning on model performance. To address this limitation, we propose PermLLM, a novel post-training pruning framework that introduces learnable channel permutation (LCP) for N:M sparsity. LCP leverages Sinkhorn normalization to transform discrete permutation matrices into differentiable soft permutation matrices, enabling end-to-end optimization. Additionally, PermLLM incorporates an efficient block-wise channel permutation strategy, which significantly reduces the number of learnable parameters and computational complexity. PermLLM seamlessly integrates with existing one-shot pruning methods to adaptively optimize channel permutations, effectively mitigating pruning-induced errors. Extensive experiments on the LLaMA series, Qwen, and OPT models demonstrate that PermLLM achieves superior performance in optimizing N:M sparse models. The code is available at https://github.com/lanchengzou/PermLLM.

Recent studies on LLM pruning primarily focus on designing a better pruning metric to obtain higher-quality masks to improve the accuracy of the sparse models [15,50,62]. RIA [62] introduces a novel pruning metric that avoids channel corruption while accounting for the effect of activations. Additionally, it proposes a two-stage channel permutation strategy to maximize the sum of retained weight importance, which serves as the quality metric to evaluate channel permutation solution. However, it is important to note that a discrepancy may exist between the handcrafted quality metric and the actual impact on output loss, as illustrated in Figure 1. Moreover, it fails to fully capture the 39th Conference on Neural Information Processing Systems (NeurIPS 2025).