ACL Findings2024
NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models
Amit Dhurandhar, Tejaswini Pedapati, Ronny Luss, Soham Dan, Aurélie C. Lozano, Payel Das, Georgios Kollias
Abstract
Transformer-based Language Models have become ubiquitous in Natural Language Processing (NLP) due to their impressive performance on various tasks. However, expensive training as well as inference remains a significant impediment to their widespread applicability. While enforcing sparsity at various levels of the model architecture has found promise in addressing scaling and efficiency issues, there remains a disconnect between how sparsity affects network topology. Inspired by brain neuronal networks, we explore sparsity approaches through the lens of network topology. Specifically, we exploit mechanisms seen in biological networks, such as preferential attachment and redundant synapse pruning, and show that principled, model-agnostic sparsity approaches are performant and efficient across diverse NLP tasks, spanning both classification (such as natural language inference) and generation (summarization, machine translation), despite our sole objective not being optimizing performance. NEUROPRUNE is competitive with (or sometimes superior to) baselines on performance and can be up to 10x faster in terms of training time for a given level of sparsity, simultaneously exhibiting measurable improvements in inference time in many cases.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Structured Pruning Learns Compact and Accurate Models cites Reducing Transformer Depth on Demand with Structured Dropout
- A Simple and Effective Pruning Approach for Large Language Models cites Structured Pruning Learns Compact and Accurate Models
- A Simple and Effective Pruning Approach for Large Language Models cites Reducing Transformer Depth on Demand with Structured Dropout
- A Simple and Effective Pruning Approach for Large Language Models cites SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot
Builds on11
- SparseGPT: Massive Language Models Can be Accurately Pruned in One-ShotElias Frantar, Dan AlistarhICML 2023 · 1,240 citations
- Pruning neural networks without any data by iteratively conserving synaptic flowHidenori Tanaka, Daniel Kunin, Daniel L. K. Yamins, Surya GanguliNeurIPS 2020 · 884 citations
- A Simple and Effective Pruning Approach for Large Language ModelsMingjie Sun, Zhuang Liu, Anna Bair, J. Zico KolterICLR 2024 · 794 citations
- Reducing Transformer Depth on Demand with Structured DropoutAngela Fan, Edouard Grave, Armand JoulinICLR 2020 · 695 citations
- Structured Pruning Learns Compact and Accurate ModelsMengzhou Xia, Zexuan Zhong, Danqi ChenACL 2022 · 236 citations
Related papers
- Sparse is Enough in Scaling TransformersSebastian Jaszczur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser et al.NeurIPS 2021 · 127 citations
- LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse ApproximationYixiao Li, Yifan Yu, Qingru Zhang, Chen Liang et al.ICML 2023 · 125 citations
- NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language ModelsJongwoo Ko, Seungjoon Park, Yujin Kim, Sumyeong Ahn et al.EMNLP Findings 2023 · 3 citations
- Structured Pruning for Efficient Generative Pre-trained Language ModelsChaofan Tao, Lu Hou, Haoli Bai, Jiansheng Wei et al.ACL Findings 2023 · 13 citations
- WINA: Weight Informed Neuron Activation for Accelerating Large Language Model InferenceSihan Chen, Dan Zhao, Jongwoo Ko, Colby Banbury et al.ICLR 2026 · 3 citations
