Lune

ICCV2023Top-tier venue

Deep Incubation: Training Large Models by Divide-and-Conquering

Zanlin Ni, Yulin Wang, Jiangwei Yu, Haojun Jiang, Yue Cao, Gao Huang

Abstract

Recent years have witnessed a remarkable success of large deep learning models. However, training these models is challenging due to high computational costs, painfully slow convergence, and overfitting issues. In this paper, we present Deep Incubation, a novel approach that enables the efficient and effective training of large models by dividing them into smaller sub-modules which can be trained separately and assembled seamlessly. A key challenge for implementing this idea is to ensure the compatibility of the independently trained sub-modules. To address this issue, we first introduce a global, shared meta model, which is leveraged to implicitly link all the modules together, and can be designed as an extremely small network with negligible computational overhead. Then we propose a module incubation algorithm, which trains each sub-module to replace the corresponding component of the meta model and accomplish a given learning task. Despite the simplicity, our approach effectively encourages each sub-module to be aware of its role in the target large model, such that the finally-learned sub-modules can collaborate with each other smoothly after being assembled. Empirically, our method outperforms end-to-end (E2E) training in terms of both final accuracy and training efficiency. For example, on top of ViT-Huge, it improves the accuracy by 2.7% on Ima-geNet or achieves similar performance with 4× less training time. Notably, the gains are significant for downstream tasks as well (e.g., object detection and image segmentation on COCO and ADE20K). Code is available at https:// github.com/LeapLabTHU/Deep-Incubation .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

7 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CoAtNet: Marrying Convolution and Attention for All Data Sizes cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • CoAtNet: Marrying Convolution and Attention for All Data Sizes cites Language Models are Few-Shot Learners
  • FLatten Transformer: Vision Transformer using Focused Linear Attention cites EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones
  • FLatten Transformer: Vision Transformer using Focused Linear Attention cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones cites A ConvNet for the 2020s
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites A Simple Framework for Contrastive Learning of Visual Representations
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
  • A ConvNet for the 2020s cites CoAtNet: Marrying Convolution and Attention for All Data Sizes
  • A ConvNet for the 2020s cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Demystify Mamba in Vision: A Linear Attention Perspective cites FLatten Transformer: Vision Transformer using Focused Linear Attention
  • Demystify Mamba in Vision: A Linear Attention Perspective cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Demystify Mamba in Vision: A Linear Attention Perspective cites A ConvNet for the 2020s
  • Demystify Mamba in Vision: A Linear Attention Perspective cites ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis
  • ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis cites Demystify Mamba in Vision: A Linear Attention Perspective
  • ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis cites FLatten Transformer: Vision Transformer using Focused Linear Attention
  • ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesis cites Language Models are Few-Shot Learners
  • GSVA: Generalized Segmentation via Multimodal Large Language Models cites EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones
  • GSVA: Generalized Segmentation via Multimodal Large Language Models cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • GSVA: Generalized Segmentation via Multimodal Large Language Models cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers7

  1. FLatten Transformer: Vision Transformer using Focused Linear AttentionDongchen Han, Xuran Pan, Yizeng Han, Shiji Song et al.ICCV 2023 · 358 citations
  2. Demystify Mamba in Vision: A Linear Attention PerspectiveDongchen Han, Ziyi Wang, Zhuofan Xia, Yizeng Han et al.NeurIPS 2024 · 287 citations
  3. GSVA: Generalized Segmentation via Multimodal Large Language ModelsZhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan et al.CVPR 2024 · 42 citations
  4. EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual BackbonesYulin Wang, Yang Yue, Rui Lu, Tianjiao Liu et al.ICCV 2023 · 39 citations
  5. ENAT: Rethinking Spatial-temporal Interactions in Token-based Image SynthesisZanlin Ni, Yulin Wang, Renping Zhou, Yizeng Han et al.NeurIPS 2024 · 15 citations

Builds on18

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  3. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  4. A ConvNet for the 2020sZhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer et al.CVPR 2022 · 6,782 citations
  5. CoAtNet: Marrying Convolution and Attention for All Data SizesZihang Dai, Hanxiao Liu, Quoc V. Le, Mingxing TanNeurIPS 2021 · 1,747 citations

Related papers

  1. Masked Autoencoders Are Scalable Vision LearnersKaiming He, Xinlei Chen, Saining Xie, Yanghao Li et al.CVPR 2022
  2. Memory Efficient Meta-Learning with Large ImagesJohn Bronskill, Daniela Massiceti, Massimiliano Patacchiola, Katja Hofmann et al.NeurIPS 2021 · 21 citations
  3. Masked Image Residual Learning for Scaling Deeper Vision TransformersGuoxi Huang, Hongtao Fu, Adrian G. BorsNeurIPS 2023 · 10 citations
  4. A Multi-Level Framework for Accelerating Training Transformer ModelsLongwei Zou, Han Zhang, Yangdong DengICLR 2024 · 3 citations
  5. MASS: MoErging through Adaptive Subspace SelectionDonato Crisostomi, Alessandro Zirilli, Antonio Andrea Gargiulo, Maria Sofia Bucarelli et al.ICLR 2026 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines