Dual Activation-Weight Sparsity: A Training-Free Framework for Efficient Large Language Model Compression

Large language models (LLMs) excel at natural language tasks but face deployment challenges due to computational demands. We introduce Dual Activation-Weight Sparsity (DAWS), a training-free framework that jointly exploits activation and weight sparsity through magnitude-based routing. Systematic analysis of pretrained transformers reveals two key observations (Figure 1 ): (1) the activation energy is concentrated in a few neurons, and (2) activation-weight contribution patterns exhibit substantial module-wise heterogeneity across attention and FFN modules. DAWS employs a three-tier routing strategy: high-magnitude activations pass through fullprecision weights to preserve critical pathways, medium-magnitude activations use magnitudepruned sparse weights for efficiency, and lowmagnitude activations are directly discarded. Unlike prior work that uses activation-aware pruning methods like WANDA, our approach uses direct magnitude-based pruning, which we show is more robust to sample-level variations. Experiments on Llama and Mistral models demonstrate that DAWS maintains >98% of dense model performance at 50% sparsity, outperforming WANDA, TEAL, and R-Sparse.