Lune

ICML2026Top-tier venue

Effective Model Pruning : Measuring the Redundancy of Model Components

Yixuan Wang, Dan Guralnik, Saiedeh Akbari, Warren Dixon

Abstract

This article initiates the study of a basic question about model pruning. Given a vector s of importance scores assigned to model components, how many of the scored components could be discarded without sacrificing performance? We propose Effective Model Pruning (EMP), which derives the desired sparsity directly from the score distribution using the notion of effective sample size from particle filtering, also known as the inverse Simpson index. Rather than prescribe a pruning criterion, EMP supplies a universal adaptive threshold derived from the distribution of the score ss over the model components: EMP maps ss to a number Neff=Neff(s)N_{eff} = N_{eff} (s), called the effective sample size. The N−NeffN − N_{eff} lowest scoring components are discarded. A tight lower bound on the preserved mass fraction seff (the sum of retained normalized scores) in terms of NeffN_{eff} is derived. This process yields models with a provable upper bound on the loss change relative to the original dense model. Numerical experiments are performed demonstrating this phenomenon across a variety of network architectures including MLPs, CNNs, Transformers, LLMs, and KAN. It is also shown that EMP addresses a rich set of pruning criteria such as weight magnitude, attention score, KAN importance score, and even feature-level signals such as image pixels.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • A Simple and Effective Pruning Approach for Large Language Models cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on11

  1. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  4. A Simple and Effective Pruning Approach for Large Language ModelsMingjie Sun, Zhuang Liu, Anna Bair, J. Zico KolterICLR 2024 · 794 citations
  5. Picking Winning Tickets Before Training by Preserving Gradient FlowChaoqi Wang, Guodong Zhang, Roger B. GrosseICLR 2020 · 743 citations

Related papers

  1. Provable Filter Pruning for Efficient Neural NetworksLucas Liebenwein, Cenk Baykal, Harry Lang, Dan Feldman et al.ICLR 2020 · 161 citations
  2. Layer-adaptive Sparsity for the Magnitude-based PruningJaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn et al.ICLR 2021 · 331 citations
  3. Data Pruning via Moving-one-Sample-outHaoru Tan, Sitong Wu, Fei Du, Yukang Chen et al.NeurIPS 2023 · 91 citations
  4. Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity PriorZi Lin, Jeremiah Z. Liu, Zi Yang, Nan Hua et al.EMNLP Findings 2020 · 34 citations
  5. What Makes a Good Prune? Maximal Unstructured Pruning for Maximal Cosine SimilarityGabryel Mason-Williams, Fredrik DahlqvistICLR 2024 · 20 citations
Dusk over the sea between two cliffs drawn in fine vertical lines