Lune

ICML2026Top-tier venue

Sparser, Faster, Lighter Transformer Language Models

Edoardo Cetin, Stefano Peluchetti, Emilio Castillo, Akira Naruse, Mana Murakami, Llion Jones

Abstract

NVIDIA, Scaling autoregressive large language models (LLMs) has driven unprecedented progress but comes with vast computational costs. In this work, we tackle these costs by leveraging unstructured sparsity within an LLM's feedforward layers, the components accounting for most of the model parameters and execution FLOPs. To achieve this, we introduce a new sparse packing format and a set of CUDA kernels designed to seamlessly integrate with the optimized execution pipelines of modern GPUs, enabling efficient sparse computation during LLM inference and training. To substantiate our gains, we provide a quantitative study of LLM sparsity, demonstrating that simple L1 regularization can induce over 99% sparsity with negligible impact on downstream performance. When paired with our kernels, we show that these sparsity levels translate into substantial throughput, energy efficiency, and memory usage benefits that increase with model scale. We will release all code and kernels under an open-source license to promote adoption and accelerate research toward establishing sparsity as a practical axis for improving the efficiency and scalability of modern foundation models.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • PIQA: Reasoning about Physical Commonsense in Natural Language cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time cites Language Models are Few-Shot Learners
  • Language Models are Few-Shot Learners cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on14

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations
  3. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations
  4. GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
  5. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference TimeZichang Liu, Jue Wang, Tri Dao, Tianyi Zhou et al.ICML 2023 · 318 citations

Related papers

  1. Learn To be Efficient: Build Structured Sparsity in Large Language ModelsHaizhong Zheng, Xiaoyan Bai, Xueshen Liu, Zhuoqing Morley Mao et al.NeurIPS 2024 · 29 citations
  2. DELTA4: Sparse Matrix-Vector Multiplication for Low SparsityVladimír Macko, Vladimír BožaICML 2026 · 9 citations
  3. SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUsRuibo Fan, Xiangrui Yu, Peijie Dong, Zeyu Li et al.EuroSys 2025 · 22 citations
  4. MECLA: Memory-Compute-Efficient LLM Accelerator with Scaling Sub-matrix PartitionYubin Qin, Yang Wang, Zhiren Zhao, Xiaolong Yang et al.ISCA 2024 · 31 citations
  5. Unified Static-Dynamic Pruning for Efficient LLM InferenceJinhyeok Kim, Yejoon Lee, Jaeyoung DoVLDB 2026
Dusk over the sea between two cliffs drawn in fine vertical lines