Lune

ASPLOS2020Top-tier venue

AutoTM: Automatic Tensor Movement in Heterogeneous Memory Systems using Integer Linear Programming

Mark Hildebrand, Jawad Khan, Sanjeev Trika, Jason Lowe-Power, Venkatesh Akella

Abstract

Memory capacity is a key bottleneck for training large scale neural networks. Intel® Optane™ DC PMM (persistent memory modules) which are available as NVDIMMs are a disruptive technology that promises significantly higher read bandwidth than traditional SSDs at a lower cost per bit than traditional DRAM. In this work we show how to take advantage of this new memory technology to minimize the amount of DRAM required without compromising performance significantly. Specifically, we take advantage of the static nature of the underlying computational graphs in deep neural network applications to develop a profile guided optimization based on Integer Linear Programming (ILP) called AutoTM to optimally assign and move live tensors to either DRAM or NVDIMMs. Our approach can replace 50% to 80% of a system's DRAM with PMM while only losing a geometric mean 27.7% performance. This is a significant improvement over first-touch NUMA, which loses 71.9% of performance. The proposed ILP based synchronous scheduling technique also provides 2x performance over using DRAM as a hardwarecontrolled cache for very large networks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

21 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers21

  1. ZeRO-Offload: Democratizing Billion-Scale Model TrainingJie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase et al.USENIX ATC 2021 · 657 citations
  2. ZeRO-infinity: breaking the GPU memory wall for extreme scale deep learningSamyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith et al.SC 2021 · 254 citations
  3. InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache ManagementWonbeom Lee, Jungi Lee, Junghwan Seo, Jaewoong SimOSDI 2024 · 248 citations
  4. CoSA: Scheduling by Constrained Optimization for Spatial AcceleratorsQijing Huang, Aravind Kalaiah, Minwoo Kang, James Demmel et al.ISCA 2021 · 120 citations
  5. MEMTIS: Efficient Memory Tiering with Dynamic Page Classification and Page Size DeterminationTaehyung Lee, Sumit Kumar Monga, Changwoo Min, Young Ik EomSOSP 2023 · 67 citations

Builds on1

  1. Single Machine Graph Analytics on Massive Datasets Using Intel Optane DC Persistent MemoryGurbinder Gill, Roshan Dathathri, Loc Hoang, Ramesh Peri et al.VLDB 2020 · 82 citations

Related papers

  1. Nap: A Black-Box Approach to NUMA-Aware Persistent Memory IndexesQing Wang, Youyou Lu, Junru Li, Jiwu ShuOSDI 2021 · 46 citations
  2. Ayudante: A Deep Reinforcement Learning Approach to Assist Persistent Memory ProgrammingHanxian Huang, Zixuan Wang, Juno Kim, Steven Swanson et al.USENIX ATC 2021 · 5 citations
  3. Understanding the Idiosyncrasies of Real Persistent MemoryShashank Gugnani, Arjun Kashyap, Xiaoyi LuVLDB 2021 · 67 citations
  4. Architecting DDR5 DRAM caches for non-volatile memory systemsXin Xin, Wanyi Zhu, Li ZhaoDAC 2022 · 3 citations
  5. PerMA-Bench: Benchmarking Persistent Memory AccessLawrence Benson, Leon Papke, Tilmann RablVLDB 2022 · 19 citations
Dusk over the sea between two cliffs drawn in fine vertical lines