Lune

NeurIPS2024Top-tier venue

Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training

Wenyu Du, Tongxu Luo, Zihan Qiu, Zeyu Huang, Yikang Shen, Reynold Cheng, Yike Guo, Jie Fu

Abstract

LLMs are computationally expensive to pre-train due to their large scale. Model growth emerges as a promising approach by leveraging smaller models to accelerate the training of larger ones. However, the viability of these model growth methods in efficient LLM pre-training remains underexplored. This work identifies three critical O‾\underline{\textit{O}}bstacles: (O\textit{O}1) lack of comprehensive evaluation, (O\textit{O}2) untested viability for scaling, and (O\textit{O}3) lack of empirical guidelines. To tackle O\textit{O}1, we summarize existing approaches into four atomic growth operators and systematically evaluate them in a standardized LLM pre-training setting. Our findings reveal that a depthwise stacking operator, called GstackG_{\text{stack}}, exhibits remarkable acceleration in training, leading to decreased loss and improved overall performance on eight standard NLP benchmarks compared to strong baselines. Motivated by these promising results, we conduct extensive experiments to delve deeper into GstackG_{\text{stack}} to address O\textit{O}2 and O\textit{O}3. For O\textit{O}2 (untested scalability), our study shows that GstackG_{\text{stack}} is scalable and consistently performs well, with experiments up to 7B LLMs after growth and pre-training LLMs with 750B tokens. For example, compared to a conventionally trained 7B model using 300B tokens, our GstackG_{\text{stack}} model converges to the same loss with 194B tokens, resulting in a 54.6% speedup. We further address O\textit{O}3 (lack of empirical guidelines) by formalizing guidelines to determine growth timing and growth factor for GstackG_{\text{stack}}, making it practical in general LLM pre-training. We also provide in-depth discussions and comprehensive ablation studies of GstackG_{\text{stack}}. Our code and pre-trained model are available at https://llm-stacking.github.io.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

13 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LESA: Learnable LLM Layer Scaling-Up cites Measuring Massive Multitask Language Understanding
  • LESA: Learnable LLM Layer Scaling-Up cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
  • LESA: Learnable LLM Layer Scaling-Up cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • LESA: Learnable LLM Layer Scaling-Up cites Language Models are Few-Shot Learners
  • SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning cites Measuring Massive Multitask Language Understanding
  • SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
  • SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning cites Language Models are Few-Shot Learners
  • Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
  • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness cites Language Models are Few-Shot Learners
  • Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Better cites Language Models are Few-Shot Learners
  • TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
  • Language Models are Few-Shot Learners cites WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers13

  1. LESA: Learnable LLM Layer Scaling-UpYifei Yang, Zouying Cao, Xinbei Ma, Yao Yao et al.ACL 2025 · 6 citations
  2. When Does Sparsity Mitigate the Curse of Depth in LLMsYao Yao, Xinyuan Song, Sebastian Pokutta, Max Zimmer et al.ICML 2026 · 5 citations
  3. SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive LearningQifan Yu, Xinyu Ma, Zhijian Zhuo, Minrui Wang et al.ICML 2026 · 3 citations
  4. From Growing to Looping: A Unified View of Iterative Computation in LLMsFerdinand Kapl, Emmanouil Angelis, Kaitlin Maile, Johannes von Oswald et al.ICML 2026 · 2 citations
  5. Late-to-Early Training: LET LLMs Learn Earlier, So Faster and BetterJi Zhao, Shitong Shao, Yufei Gu, Xun Zhou et al.ICLR 2026 · 1 citation

Builds on20

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra et al.NeurIPS 2022 · 5,493 citations
  4. TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
  5. WinoGrande: An Adversarial Winograd Schema Challenge at ScaleKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin ChoiAAAI 2020 · 3,037 citations

Related papers

  1. Masked Structural Growth for 2x Faster Language Model Pre-trainingYiqun Yao, Zheng Zhang, Jing Li, Yequan WangICLR 2024 · 30 citations
  2. Deep Fusion: Efficient Network Training via Pre-trained InitializationsHanna Mazzawi, Javier Gonzalvo, Michael Wunder, Sammy Jerome et al.ICML 2024 · 4 citations
  3. On the Inductive Bias of Stacking Towards Improving ReasoningNikunj Saunshi, Stefani Karp, Shankar Krishnan, Sobhan Miryoosefi et al.NeurIPS 2024 · 23 citations
  4. Scaling depth capacity via zero/one-layer model expansionZhiqi BuICML 2026 · 1 citation
  5. Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNPengxiang Li, Lu Yin, Shiwei LiuICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines