AutoTM: Automatic Tensor Movement in Heterogeneous Memory Systems using Integer Linear Programming
Mark Hildebrand, Jawad Khan, Sanjeev Trika, Jason Lowe-Power, Venkatesh Akella
Abstract
Memory capacity is a key bottleneck for training large scale neural networks. Intel® Optane™ DC PMM (persistent memory modules) which are available as NVDIMMs are a disruptive technology that promises significantly higher read bandwidth than traditional SSDs at a lower cost per bit than traditional DRAM. In this work we show how to take advantage of this new memory technology to minimize the amount of DRAM required without compromising performance significantly. Specifically, we take advantage of the static nature of the underlying computational graphs in deep neural network applications to develop a profile guided optimization based on Integer Linear Programming (ILP) called AutoTM to optimally assign and move live tensors to either DRAM or NVDIMMs. Our approach can replace 50% to 80% of a system's DRAM with PMM while only losing a geometric mean 27.7% performance. This is a significant improvement over first-touch NUMA, which loses 71.9% of performance. The proposed ILP based synchronous scheduling technique also provides 2x performance over using DRAM as a hardwarecontrolled cache for very large networks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
21 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers21
- ZeRO-Offload: Democratizing Billion-Scale Model TrainingJie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase et al.USENIX ATC 2021 · 657 citations
- ZeRO-infinity: breaking the GPU memory wall for extreme scale deep learningSamyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith et al.SC 2021 · 254 citations
- InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache ManagementWonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong SimOSDI 2024 · 248 citations
- CoSA: Scheduling by Constrained Optimization for Spatial AcceleratorsQijing Huang, Aravind Kalaiah, Minwoo Kang, James Demmel et al.ISCA 2021 · 120 citations
- MEMTIS: Efficient Memory Tiering with Dynamic Page Classification and Page Size DeterminationTaehyung Lee, Sumit Kumar Monga, Changwoo Min, Young Ik EomSOSP 2023 · 67 citations
Builds on1
Related papers
- Nap: A Black-Box Approach to NUMA-Aware Persistent Memory IndexesQing Wang, Youyou Lu, Junru Li, Jiwu ShuOSDI 2021 · 46 citations
- Ayudante: A Deep Reinforcement Learning Approach to Assist Persistent Memory ProgrammingHanxian Huang, Zixuan Wang, Juno Kim, Steven Swanson et al.USENIX ATC 2021 · 5 citations
- Understanding the Idiosyncrasies of Real Persistent MemoryShashank Gugnani, Arjun Kashyap, Xiaoyi LuVLDB 2021 · 67 citations
- Architecting DDR5 DRAM caches for non-volatile memory systemsXin Xin, Wanyi Zhu, Li ZhaoDAC 2022 · 3 citations
- PerMA-Bench: Benchmarking Persistent Memory AccessLawrence Benson, Leon Papke, Tilmann RablVLDB 2022 · 19 citations
