Activation Map Compression through Tensor Decomposition for Deep Learning
Le-Trung Nguyen, Aël Quélennec, Enzo Tartaglione, Samuel Tardieu, Van-Tam Nguyen
Abstract
Internet of Things and Deep Learning are synergetically and exponentially growing industrial fields with a massive call for their unification into a common framework called Edge AI. While on-device inference is a well-explored topic in recent research, backpropagation remains an open challenge due to its prohibitive computational and memory costs compared to the extreme resource constraints of embedded devices. Drawing on tensor decomposition research, we tackle the main bottleneck of backpropagation, namely the memory footprint of activation map storage. We investigate and compare the effects of activation compression using Singular Value Decomposition and its tensor variant, High-Order Singular Value Decomposition. The application of low-order decomposition results in considerable memory savings while preserving the features essential for learning, and also offers theoretical guarantees to convergence. Experimental results obtained on main-stream architectures and tasks demonstrate Pareto-superiority over other state-of-the-art solutions, in terms of the trade-off between generalization and memory footprint.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- INSTANT: Compressing Gradients and Activations for Resource-Efficient Training cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- Efficient Resource-Constrained Training of Transformers via Subspace Optimization cites On-Device Training Under 256KB Memory
- Efficient Resource-Constrained Training of Transformers via Subspace Optimization cites Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning
- Efficient Resource-Constrained Training of Transformers via Subspace Optimization cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- On-Device Training Under 256KB Memory cites MCUNet: Tiny Deep Learning on IoT Devices
- Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning cites On-Device Training Under 256KB Memory
- Study of Training Dynamics for Memory-Constrained Fine-Tuning cites On-Device Training Under 256KB Memory
- Study of Training Dynamics for Memory-Constrained Fine-Tuning cites MCUNet: Tiny Deep Learning on IoT Devices
- Study of Training Dynamics for Memory-Constrained Fine-Tuning cites Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning
- Study of Training Dynamics for Memory-Constrained Fine-Tuning cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- Study of Training Dynamics for Memory-Constrained Fine-Tuning cites TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning
Cited by top-tier papers5
- Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device LearningLe-Trung Nguyen, Aël Quélennec, Van-Tam Nguyen, Enzo TartaglioneICML 2025 · 6 citations
- Study of Training Dynamics for Memory-Constrained Fine-TuningAël Quélennec, Nour Hezbri, Pavlo Mozharovskyi, Van-Tam Nguyen et al.ICLR 2026 · 1 citation
- INSTANT: Compressing Gradients and Activations for Resource-Efficient TrainingTuan-Kiet Doan, Trung-Hieu Tran, Enzo Tartaglione, Nikola Simidjievski et al.ICLR 2026
- Efficient Resource-Constrained Training of Transformers via Subspace OptimizationLe-Trung Nguyen, Enzo Tartaglione, Van-Tam NguyenICLR 2026
- MTNL: A Unified Modeling Perspective for Enhancing Tensor Network LearningJunhua Zeng, Yuning Qiu, Binghua Li, Chao Li et al.ICML 2026
Builds on10
- Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
- MCUNet: Tiny Deep Learning on IoT DevicesJi Lin, Wei-Ming Chen, Yujun Lin, John Cohn et al.NeurIPS 2020 · 827 citations
- TinyTL: Reduce Memory, Not Parameters for Efficient On-Device LearningHan Cai, Chuang Gan, Ligeng Zhu, Song HanNeurIPS 2020 · 375 citations
- On-Device Training Under 256KB MemoryJi Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang et al.NeurIPS 2022 · 345 citations
- Inducing and Exploiting Activation Sparsity for Fast Inference on Deep Neural NetworksMark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev et al.ICML 2020 · 163 citations
Related papers
- Convolutional Neural Network Compression through Generalized Kronecker Product DecompositionMarawan Gamal Abdel Hameed, Marzieh S. Tahaei, Ali Mosleh, Vahid Partovi NiaAAAI 2022 · 33 citations
- FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank ModelsZishan Shao, Yixiao Wang, Qinsi Wang, Ting Jiang et al.AAAI 2026 · 2 citations
- TinyFoA: Memory Efficient Forward-Only Algorithm for On-Device LearningBaichuan Huang, Amir AminifarAAAI 2025 · 3 citations
- MST-compression: Compressing and Accelerating Binary Neural Networks with Minimum Spanning TreeQuang Hieu Vo, Linh-Tam Tran, Sung-Ho Bae, Lok-Won Kim et al.ICCV 2023 · 2 citations
- ALF: Autoencoder-based Low-rank Filter-sharing for Efficient Convolutional Neural NetworksAlexander Frickenstein, Manoj Rohit Vemparala, Nael Fasfous, Laura Hauenschild et al.DAC 2020 · 5 citations
