SC2022Top-tier venue
HammingMesh: A Network Topology for Large-Scale Deep Learning
Torsten Hoefler, Tommaso Bonato, Daniele De Sensi, Salvatore Di Girolamo, Shigang Li, Marco Heddes, Jon Belk, Deepak Goel, Miguel Castro, Steve Scott
Abstract
Numerous microarchitectural optimizations unlocked tremendous processing power for deep neural networks that in turn fueled the AI revolution. With the exhaustion of such optimizations, the growth of modern AI is now gated by the performance of training systems, especially their data movement. Instead of focusing on single accelerators, we investigate data-movement characteristics of large-scale training at full system scale. Based on our workload analysis, we design HammingMesh, a novel network topology that provides high bandwidth at low cost with high job scheduling flexibility. Specifically, HammingMesh can support full bandwidth and isolation to deep learning training jobs with two dimensions of parallelism. Furthermore, it also supports high global bandwidth for generic traffic. Thus, HammingMesh will power future large-scale deep learning systems with extreme bandwidth requirements.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
15 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Switch-Less Dragonfly on Wafers: A Scalable Interconnection Architecture based on Wafer-Scale Integration cites Swing: Short-cutting Rings for Higher Bandwidth Allreduce
- Switch-Less Dragonfly on Wafers: A Scalable Interconnection Architecture based on Wafer-Scale Integration cites Evaluating Chiplet-based Large-Scale Interconnection Networks via Cycle-Accurate Packet-Parallel Simulation
- A High-Performance Design, Implementation, Deployment, and Evaluation of The Slim Fly Network cites Language Models are Few-Shot Learners
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding cites Language Models are Few-Shot Learners
- Chimera: efficiently training large-scale neural networks with bidirectional pipelines cites ZeRO: memory optimizations toward training trillion parameter models
- Chimera: efficiently training large-scale neural networks with bidirectional pipelines cites Language Models are Few-Shot Learners
Cited by top-tier papers15
- Swing: Short-cutting Rings for Higher Bandwidth AllreduceDaniele De Sensi, Tommaso Bonato, David Saam, Torsten HoeflerNSDI 2024 · 48 citations
- Evaluating Chiplet-based Large-Scale Interconnection Networks via Cycle-Accurate Packet-Parallel SimulationYinxiao Feng, Yuchen Wei, Dong Xiang, Kaisheng MaUSENIX ATC 2024 · 21 citations
- A High-Performance Design, Implementation, Deployment, and Evaluation of The Slim Fly NetworkNils Blach, Maciej Besta, Daniele De Sensi, Jens Domke et al.NSDI 2024 · 13 citations
- REPS: Recycled Entropy Packet Spraying for Adaptive Load Balancing and Failure MitigationTommaso Bonato, Abdul Kabbani, Ahmad Ghalayini, Michael Papamichael et al.EuroSys 2026 · 12 citations
- Switch-Less Dragonfly on Wafers: A Scalable Interconnection Architecture based on Wafer-Scale IntegrationYinxiao Feng, Kaisheng MaSC 2024 · 10 citations
Builds on13
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
- ZeRO: memory optimizations toward training trillion parameter modelsSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong HeSC 2020 · 852 citations
- Pushing the Limits of Narrow Precision Inferencing at Cloud Scale with Microsoft Floating PointBita Darvish Rouhani, Daniel Lo, Ritchie Zhao, Ming Liu et al.NeurIPS 2020 · 153 citations
- Chimera: efficiently training large-scale neural networks with bidirectional pipelinesShigang Li, Torsten HoeflerSC 2021 · 124 citations
Related papers
- FRED: A Wafer-scale Fabric for 3D Parallel DNN TrainingSaeed Rashidi, William Won, Sudarshan Srinivasan, Puneet Gupta et al.ISCA 2025 · 8 citations
- Communication Algorithm-Architecture Co-Design for Distributed Deep LearningJiayi Huang, Pritam Majumder, Sungkeun Kim, Abdullah Muzahid et al.ISCA 2021 · 44 citations
- PD Constraint-aware Physical/Logical Topology Co-Design for Network on WaferQize Yang, Taiquan Wei, Sihan Guan, Chengran Li et al.ISCA 2025 · 16 citations
- Near-Optimal Topology-adaptive Parameter Synchronization in Distributed DNN TrainingZhe Zhang, Chuan Wu, Zongpeng LiINFOCOM 2021 · 14 citations
- TrainBox: An Extreme-Scale Neural Network Training Server Architecture by Systematically Balancing OperationsPyeongsu Park, Heetaek Jeong, Jangwoo KimMICRO 2020 · 11 citations
