Lune

OSDI2022Top-tier venue

Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

Lianmin Zheng, Zhuohan Li, Hao Zhang, Yonghao Zhuang, Zhifeng Chen, Yanping Huang, Yida Wang, Yuanzhong Xu, Danyang Zhuo, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica

Abstract

Alpa automates model-parallel training of large deep learning (DL) models by generating execution plans that unify data, operator, and pipeline parallelism. Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations. They do not suffice to scale out complex DL models on distributed compute devices. Alpa distributes the training of large DL models by viewing parallelisms as two hierarchical levels: inter-operator and intra-operator parallelisms. Based on it, Alpa constructs a new hierarchical space for massive model-parallel execution plans. Alpa designs a number of compilation passes to automatically derive efficient parallel execution plans at each parallelism level. Alpa implements an efficient runtime to orchestrate the two-level parallel execution on distributed compute devices. Our evaluation shows Alpa generates parallelization plans that match or outperform hand-tuned model-parallel training systems even on models they are designed for. Unlike specialized systems, Alpa also generalizes to models with heterogeneous architectures and models without manuallydesigned plans. Alpa's source code is publicly available at https://github.com/alpa-projects/alpa .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

158 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving cites ZeRO: memory optimizations toward training trillion parameter models
  • AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving cites Efficient large-scale language model training on GPU clusters using megatron-LM
  • AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving cites Language Models are Few-Shot Learners
  • AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
  • Efficient large-scale language model training on GPU clusters using megatron-LM cites ZeRO: memory optimizations toward training trillion parameter models
  • Efficient large-scale language model training on GPU clusters using megatron-LM cites Language Models are Few-Shot Learners
  • Efficient large-scale language model training on GPU clusters using megatron-LM cites ZeRO-Offload: Democratizing Billion-Scale Model Training
  • FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU cites Efficient large-scale language model training on GPU clusters using megatron-LM
  • FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU cites Language Models are Few-Shot Learners
  • FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU cites ZeRO-Offload: Democratizing Billion-Scale Model Training
  • Efficient Memory Management for Large Language Model Serving with PagedAttention cites AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
  • Efficient Memory Management for Large Language Model Serving with PagedAttention cites FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
  • Efficient Memory Management for Large Language Model Serving with PagedAttention cites Language Models are Few-Shot Learners
  • Efficient Memory Management for Large Language Model Serving with PagedAttention cites ZeRO-Offload: Democratizing Billion-Scale Model Training
  • DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving cites AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
  • DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving cites ZeRO: memory optimizations toward training trillion parameter models
  • DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving cites Efficient Memory Management for Large Language Model Serving with PagedAttention
  • GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding cites Language Models are Few-Shot Learners
  • ZeRO-Offload: Democratizing Billion-Scale Model Training cites ZeRO: memory optimizations toward training trillion parameter models
  • Characterization of Large Language Model Development in the Datacenter cites AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
  • Characterization of Large Language Model Development in the Datacenter cites ZeRO: memory optimizations toward training trillion parameter models
  • Characterization of Large Language Model Development in the Datacenter cites Efficient large-scale language model training on GPU clusters using megatron-LM
  • Characterization of Large Language Model Development in the Datacenter cites Efficient Memory Management for Large Language Model Serving with PagedAttention
  • Characterization of Large Language Model Development in the Datacenter cites Language Models are Few-Shot Learners
  • Characterization of Large Language Model Development in the Datacenter cites ZeRO-Offload: Democratizing Billion-Scale Model Training
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers158

  1. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
  2. FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUYing Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li et al.ICML 2023 · 683 citations
  3. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model ServingYinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu et al.OSDI 2024 · 646 citations
  4. AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning ServingZhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu et al.OSDI 2023 · 211 citations
  5. Characterization of Large Language Model Development in the DatacenterQinghao Hu, Zhisheng Ye, Zerui Wang, Guoteng Wang et al.NSDI 2024 · 192 citations

Builds on19

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
  3. ZeRO: memory optimizations toward training trillion parameter modelsSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong HeSC 2020 · 852 citations
  4. ZeRO-Offload: Democratizing Billion-Scale Model TrainingJie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase et al.USENIX ATC 2021 · 657 citations
  5. Efficient large-scale language model training on GPU clusters using megatron-LMDeepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley et al.SC 2021 · 576 citations

Related papers

  1. nnScaler: Constraint-Guided Parallelization Plan Generation for Deep Learning TrainingZhiqi Lin, Youshan Miao, Quanlu Zhang, Fan Yang et al.OSDI 2024 · 38 citations
  2. Metis: Fast Automatic Distributed Training on Heterogeneous GPUsTaegeon Um, Byungsoo Oh, Minyoung Kang, Woo-Yeon Lee et al.USENIX ATC 2024 · 81 citations
  3. Efficient Algorithms for Device Placement of DNN Graph OperatorsJakub Tarnawski, Amar Phanishayee, Nikhil R. Devanur, Divya Mahajan et al.NeurIPS 2020 · 84 citations
  4. UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic ProgrammingHao Lin, Ke Wu, Jie Li, Jun Li et al.CVPR 2025
  5. Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data AnnotationsHaoyang Li, Fangcheng Fu, Hao Ge, Sheng Lin et al.OSDI 2026 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines