EfficientNetV2: Smaller Models and Faster Training
Mingxing Tan, Quoc V. Le
Abstract
This paper introduces EfficientNetV2, a new family of convolutional networks that have faster training speed and better parameter efficiency than previous models. To develop this family of models, we use a combination of training-aware neural architecture search and scaling, to jointly optimize training speed and parameter efficiency. The models were searched from the search space enriched with new ops such as Fused-MBConv. Our experiments show that EfficientNetV2 models train much faster than state-of-the-art models while being up to 6.8x smaller. Our training can be further sped up by progressively increasing the image size during training, but it often causes a drop in accuracy. To compensate for this accuracy drop, we propose to adaptively adjust regularization (e.g., dropout and data augmentation) as well, such that we can achieve both fast training and good accuracy. With progressive learning, our EfficientNetV2 significantly outperforms previous models on ImageNet and CIFAR/Cars/Flowers datasets. By pretraining on the same ImageNet21k, our EfficientNetV2 achieves 87.3% top-1 accuracy on ImageNet ILSVRC2012, outperforming the recent ViT by 2.0% accuracy while training 5x-11x faster using the same computing resources. Code will be available at https://github.com/google/automl/tree/master/efficientnetv2.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
192 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- CoAtNet: Marrying Convolution and Attention for All Data Sizes cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- CoAtNet: Marrying Convolution and Attention for All Data Sizes cites RandAugment: Practical Automated Data Augmentation with a Reduced Search Space
- CoAtNet: Marrying Convolution and Attention for All Data Sizes cites LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
- CoAtNet: Marrying Convolution and Attention for All Data Sizes cites Language Models are Few-Shot Learners
- CoAtNet: Marrying Convolution and Attention for All Data Sizes cites Training data-efficient image transformers & distillation through attention
- CoAtNet: Marrying Convolution and Attention for All Data Sizes cites Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
- Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model cites CoAtNet: Marrying Convolution and Attention for All Data Sizes
- Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model cites A ConvNet for the 2020s
- Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model cites Training data-efficient image transformers & distillation through attention
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
- A ConvNet for the 2020s cites CoAtNet: Marrying Convolution and Attention for All Data Sizes
- A ConvNet for the 2020s cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- A ConvNet for the 2020s cites RandAugment: Practical Automated Data Augmentation with a Reduced Search Space
- A ConvNet for the 2020s cites Training data-efficient image transformers & distillation through attention
- A ConvNet for the 2020s cites Early Convolutions Help Transformers See Better
- LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference cites Training data-efficient image transformers & distillation through attention
- LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference cites Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
- Training data-efficient image transformers & distillation through attention cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Early Convolutions Help Transformers See Better cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Early Convolutions Help Transformers See Better cites LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
- Early Convolutions Help Transformers See Better cites Training data-efficient image transformers & distillation through attention
- Early Convolutions Help Transformers See Better cites Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
- Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet cites Language Models are Few-Shot Learners
- Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet cites Training data-efficient image transformers & distillation through attention
Cited by top-tier papers192
- A ConvNet for the 2020sZhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer et al.CVPR 2022 · 6,782 citations
- CoAtNet: Marrying Convolution and Attention for All Data SizesZihang Dai, Hanxiao Liu, Quoc V. Le, Mingxing TanNeurIPS 2021 · 1,747 citations
- Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space ModelLianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang et al.ICML 2024 · 1,725 citations
- LeViT: a Vision Transformer in ConvNet's Clothing for Faster InferenceBenjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock et al.ICCV 2021 · 1,009 citations
- Early Convolutions Help Transformers See BetterTete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell et al.NeurIPS 2021 · 974 citations
Builds on11
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Training data-efficient image transformers & distillation through attentionHugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa et al.ICML 2021 · 8,974 citations
- RandAugment: Practical Automated Data Augmentation with a Reduced Search SpaceEkin Dogus Cubuk, Barret Zoph, Jonathon Shlens, Quoc LeNeurIPS 2020 · 4,453 citations
- Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetLi Yuan, Yunpeng Chen, Tao Wang, Weihao Yu et al.ICCV 2021 · 2,462 citations
Related papers
- Automated Progressive Learning for Efficient Training of Vision TransformersChanglin Li, Bohan Zhuang, Guangrun Wang, Xiaodan Liang et al.CVPR 2022 · 28 citations
- Wisdom of Committees: An Overlooked Approach To Faster and More Accurate ModelsXiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M. Kitani et al.ICLR 2022 · 61 citations
- Searching for Fast Model Families on Datacenter AcceleratorsSheng Li, Mingxing Tan, Ruoming Pang, Andrew Li et al.CVPR 2021
- Auto-scaling Vision Transformers without TrainingWuyang Chen, Wei Huang, Xianzhi Du, Xiaodan Song et al.ICLR 2022 · 27 citations
- FP-NAS: Fast Probabilistic Neural Architecture SearchZhicheng Yan, Xiaoliang Dai, Peizhao Zhang, Yuandong Tian et al.CVPR 2021
