Lune

CVPR2021Top-tier venue

NetAdaptV2: Efficient Neural Architecture Search With Fast Super-Network Training and Architecture Optimization

Tien-Ju Yang, Yi-Lun Liao, Vivienne Sze

Abstract

Neural architecture search (NAS) typically consists of three main steps: training a super-network, training and evaluating sampled deep neural networks (DNNs), and training the discovered DNN. Most of the existing efforts speed up some steps at the cost of a significant slowdown of other steps or sacrificing the support of non-differentiable search metrics. The unbalanced reduction in the time spent per step limits the total search time reduction, and the inability to support non-differentiable search metrics limits the performance of discovered DNNs. In this paper, we present NetAdaptV2 with three innovations to better balance the time spent for each step while supporting non-differentiable search metrics. First, we propose channel-level bypass connections that merge network depth and layer width into a single search dimension to reduce the time for training and evaluating sampled DNNs. Second, ordered dropout is proposed to train multiple DNNs in a single forward-backward pass to decrease the time for training a super-network. Third, we propose the multi-layer coordinate descent optimizer that considers the interplay of multiple layers in each iteration of optimization to improve the performance of discovered DNNs while supporting non-differentiable search metrics. With these innovations, NetAdaptV2 reduces the total search time by up to 5.8× on ImageNet and 2.4× on NYU Depth V2, respectively, and discovers DNNs with better accuracy-latency/accuracy-MAC trade-offs than state-of-the-art NAS works. Moreover, the discovered DNN outperforms NAS-discovered MobileNetV3 by 1.8% higher top-1 accuracy with the same latency. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SPDY: Accurate Pruning with Speedup Guarantees cites Once-for-All: Train One Network and Specialize it for Efficient Deployment
  • FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search cites Searching for MobileNetV3
  • FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search cites PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
  • FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search cites Once-for-All: Train One Network and Specialize it for Efficient Deployment
  • Once-for-All: Train One Network and Specialize it for Efficient Deployment cites Universally Slimmable Networks and Improved Training Techniques
  • Once-for-All: Train One Network and Specialize it for Efficient Deployment cites Searching for MobileNetV3
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. SPDY: Accurate Pruning with Speedup GuaranteesElias Frantar, Dan AlistarhICML 2022 · 45 citations

Builds on9

  1. Searching for MobileNetV3Andrew Howard, Ruoming Pang, Hartwig Adam, Quoc V. Le et al.ICCV 2019 · 9,163 citations
  2. Once-for-All: Train One Network and Specialize it for Efficient DeploymentHan Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang et al.ICLR 2020 · 1,522 citations
  3. PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture SearchYuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen et al.ICLR 2020 · 691 citations
  4. Universally Slimmable Networks and Improved Training TechniquesJiahui Yu, Thomas S. HuangICCV 2019 · 444 citations
  5. FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture SearchXiangxiang Chu, Bo Zhang, Ruijun XuICCV 2021 · 362 citations

Related papers

  1. FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel DimensionsAlvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He et al.CVPR 2020
  2. Fast and Practical Neural Architecture SearchJiequan Cui, Pengguang Chen, Ruiyu Li, Shu Liu et al.ICCV 2019 · 69 citations
  3. FP-NAS: Fast Probabilistic Neural Architecture SearchZhicheng Yan, Xiaoliang Dai, Peizhao Zhang, Yuandong Tian et al.CVPR 2021
  4. UNAS: Differentiable Architecture Search Meets Reinforcement LearningArash Vahdat, Arun Mallya, Ming-Yu Liu, Jan KautzCVPR 2020
  5. ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse CodingYibo Yang, Hongyang Li, Shan You, Fei Wang et al.NeurIPS 2020 · 66 citations
Dusk over the sea between two cliffs drawn in fine vertical lines