NetAdaptV2: Efficient Neural Architecture Search With Fast Super-Network Training and Architecture Optimization
Tien-Ju Yang, Yi-Lun Liao, Vivienne Sze
Abstract
Neural architecture search (NAS) typically consists of three main steps: training a super-network, training and evaluating sampled deep neural networks (DNNs), and training the discovered DNN. Most of the existing efforts speed up some steps at the cost of a significant slowdown of other steps or sacrificing the support of non-differentiable search metrics. The unbalanced reduction in the time spent per step limits the total search time reduction, and the inability to support non-differentiable search metrics limits the performance of discovered DNNs. In this paper, we present NetAdaptV2 with three innovations to better balance the time spent for each step while supporting non-differentiable search metrics. First, we propose channel-level bypass connections that merge network depth and layer width into a single search dimension to reduce the time for training and evaluating sampled DNNs. Second, ordered dropout is proposed to train multiple DNNs in a single forward-backward pass to decrease the time for training a super-network. Third, we propose the multi-layer coordinate descent optimizer that considers the interplay of multiple layers in each iteration of optimization to improve the performance of discovered DNNs while supporting non-differentiable search metrics. With these innovations, NetAdaptV2 reduces the total search time by up to 5.8× on ImageNet and 2.4× on NYU Depth V2, respectively, and discovers DNNs with better accuracy-latency/accuracy-MAC trade-offs than state-of-the-art NAS works. Moreover, the discovered DNN outperforms NAS-discovered MobileNetV3 by 1.8% higher top-1 accuracy with the same latency. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- SPDY: Accurate Pruning with Speedup Guarantees cites Once-for-All: Train One Network and Specialize it for Efficient Deployment
- FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search cites Searching for MobileNetV3
- FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search cites PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search cites Once-for-All: Train One Network and Specialize it for Efficient Deployment
- Once-for-All: Train One Network and Specialize it for Efficient Deployment cites Universally Slimmable Networks and Improved Training Techniques
- Once-for-All: Train One Network and Specialize it for Efficient Deployment cites Searching for MobileNetV3
Cited by top-tier papers1
Builds on9
- Searching for MobileNetV3Andrew Howard, Ruoming Pang, Hartwig Adam, Quoc V. Le et al.ICCV 2019 · 9,163 citations
- Once-for-All: Train One Network and Specialize it for Efficient DeploymentHan Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang et al.ICLR 2020 · 1,522 citations
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture SearchYuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen et al.ICLR 2020 · 691 citations
- Universally Slimmable Networks and Improved Training TechniquesJiahui Yu, Thomas S. HuangICCV 2019 · 444 citations
- FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture SearchXiangxiang Chu, Bo Zhang, Ruijun XuICCV 2021 · 362 citations
Related papers
- FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel DimensionsAlvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He et al.CVPR 2020
- Fast and Practical Neural Architecture SearchJiequan Cui, Pengguang Chen, Ruiyu Li, Shu Liu et al.ICCV 2019 · 69 citations
- FP-NAS: Fast Probabilistic Neural Architecture SearchZhicheng Yan, Xiaoliang Dai, Peizhao Zhang, Yuandong Tian et al.CVPR 2021
- UNAS: Differentiable Architecture Search Meets Reinforcement LearningArash Vahdat, Arun Mallya, Ming-Yu Liu, Jan KautzCVPR 2020
- ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse CodingYibo Yang, Hongyang Li, Shan You, Fei Wang et al.NeurIPS 2020 · 66 citations
