Lune

CVPR Findings2026

Debiased One-Shot NAS Via Density-Aware Sampling

Mehraveh Javan Roshtkhari, Matthew Toews, Marco Pedersoli

Abstract

One-shot Neural Architecture Search (NAS) is based on training a supernet, a single model from which many different architectures with shared weights are sampled and updated during training. Training the shared weights of the supernet is much more computationally efficient than training each architecture independently. However, during the supernet training, architectures with similar gradients (dense regions in the gradient space) would cooperate with each other, increasing their training, while architectures in sparse (low-density) regions would not receive as much training benefit from others. This does not allow all architectures to be trained with the same amount of effective updates, resulting in a training bias that favors architectures in denser regions. As a consequence, the correlation between the supernet estimations and the actual performance of models independently trained is reduced. This negatively affects the supernet's ability to select good architectures once trained. In this paper, we propose two computationally feasible ways for different computational budgets and search spaces to approximate the architecture densities and implement a density-aware debiasing mechanism for supernet training. We propose a fine-grained density calculation for numerable smaller search spaces and an online density approximation based on density prototypes from a clustering algorithm for larger spaces. We validate our method on CIFAR10, CIFAR100, and ImageNet datasets using various search strategies and show consistently improved results compared to several singlepath one-shot supernet training methods.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search cites Evaluating The Search Phase of Neural Architecture Search
  • Evaluating The Search Phase of Neural Architecture Search cites FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search
  • FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search cites Evaluating The Search Phase of Neural Architecture Search
  • FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search cites NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
  • FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search cites Once-for-All: Train One Network and Specialize it for Efficient Deployment
  • FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search cites Understanding and Robustifying Differentiable Architecture Search
  • Understanding and Robustifying Differentiable Architecture Search cites Evaluating The Search Phase of Neural Architecture Search
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on18

  1. Once-for-All: Train One Network and Specialize it for Efficient DeploymentHan Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang et al.ICLR 2020 · 1,522 citations
  2. NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture SearchXuanyi Dong, Yi YangICLR 2020 · 825 citations
  3. Understanding and Robustifying Differentiable Architecture SearchArber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi et al.ICLR 2020 · 408 citations
  4. Evaluating The Search Phase of Neural Architecture SearchKaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat et al.ICLR 2020 · 370 citations
  5. FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture SearchXiangxiang Chu, Bo Zhang, Ruijun XuICCV 2021 · 362 citations

Related papers

  1. Distribution Consistent Neural Architecture SearchJunyi Pan, Chong Sun, Yizhou Zhou, Ying Zhang et al.CVPR 2022 · 9 citations
  2. SUMNAS: Supernet with Unbiased Meta-Features for Neural Architecture SearchHyeonmin Ha, Ji-Hoon Kim, Semin Park, Byung-Gon ChunICLR 2022 · 5 citations
  3. ShiftNAS: Improving One-shot NAS via Probability ShiftMingyang Zhang, Xinyi Yu, Haodong Zhao, Linlin OuICCV 2023 · 9 citations
  4. PA&DA: Jointly Sampling PAth and DAta for Consistent NASShun Lu, Yu Hu, Longxing Yang, Zihao Sun et al.CVPR 2023
  5. Few-Shot Neural Architecture SearchYiyang Zhao, Linnan Wang, Yuandong Tian, Rodrigo Fonseca et al.ICML 2021 · 100 citations
Dusk over the sea between two cliffs drawn in fine vertical lines