Lune

ACL Findings2024

Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts

Ganesh Jawahar, Haichuan Yang, Yunyang Xiong, Zechun Liu, Dilin Wang, Fei Sun, Meng Li, Aasish Pappu, Barlas Oguz, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Raghuraman Krishnamoorthi, Vikas Chandra

Abstract

Weight-sharing supernets are crucial for performance estimation in cutting-edge neural architecture search (NAS) frameworks. Despite their ability to generate diverse subnetworks without retraining, the quality of these subnetworks is not guaranteed due to weight sharing. In NLP tasks like machine translation and pre-trained language modeling, there is a significant performance gap between supernet and training from scratch for the same model architecture, necessitating retraining post optimal architecture identification.

This study introduces a solution called mixtureof-supernets, a generalized supernet formulation leveraging mixture-of-experts (MoE) to enhance supernet model expressiveness with minimal training overhead. Unlike conventional supernets, this method employs an architecturebased routing mechanism, enabling indirect sharing of model weights among subnetworks. This customization of weights for specific architectures, learned through gradient descent, minimizes retraining time, significantly enhancing training efficiency in NLP. The proposed method attains state-of-the-art (SoTA) performance in NAS for fast machine translation models, exhibiting a superior latency-BLEU tradeoff compared to HAT, the SoTA NAS framework for machine translation. Furthermore, it excels in NAS for building memoryefficient task-agnostic BERT models, surpassing NAS-BERT and AutoDistil across various model sizes. The code can be found at: https://github.com/UBC-NLP/MoS.

  • Some of the work was completed while Ganesh was interning at Meta.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TinyBERT: Distilling BERT for Natural Language Understanding cites MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
  • TAS-LoRA: Transformer Architecture Search with Mixture-of-LoRA Experts cites Once-for-All: Train One Network and Specialize it for Efficient Deployment
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use CasesZechun Liu, Changsheng Zhao, Forrest N. Iandola, Chen Lai et al.ICML 2024 · 227 citations
  2. Mixture of Mini Experts: Overcoming the Linear Layer Bottleneck in Multiple Instance LearningDaniel Shao, Joel Runevic, Richard J. Chen, Drew F. K. Williamson et al.ICLR 2026 · 3 citations
  3. TAS-LoRA: Transformer Architecture Search with Mixture-of-LoRA ExpertsJeimin Jeon, Hyunju Lee, Bumsub HamCVPR 2026

Builds on9

  1. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained TransformersWenhui Wang, Furu Wei, Li Dong, Hangbo Bao et al.NeurIPS 2020 · 2,727 citations
  2. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  3. Once-for-All: Train One Network and Specialize it for Efficient DeploymentHan Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang et al.ICLR 2020 · 1,522 citations
  4. HAT: Hardware-Aware Transformers for Efficient Natural Language ProcessingHanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai et al.ACL 2020 · 215 citations
  5. NASViT: Neural Architecture Search for Efficient Vision Transformers with Gradient Conflict aware Supernet TrainingChengyue Gong, Dilin Wang, Meng Li, Xinlei Chen et al.ICLR 2022 · 114 citations

Related papers

  1. AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine TranslationGanesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu, Young Jin Kim et al.ACL Findings 2023 · 5 citations
  2. Weight-Sharing NAS with Architecture-Agnostic Intermediate RepresentationSixing Yu, Arya Mazaheri, Ali JannesariHPDC 2025
  3. SUMNAS: Supernet with Unbiased Meta-Features for Neural Architecture SearchHyeonmin Ha, Ji-Hoon Kim, Semin Park, Byung-Gon ChunICLR 2022 · 5 citations
  4. Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMsZhongyang Li, Ziyue Li, Tianyi ZhouICLR 2026 · 5 citations
  5. Beyond Distillation: Task-level Mixture-of-Experts for Efficient InferenceSneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun et al.EMNLP Findings 2021 · 41 citations
Dusk over the sea between two cliffs drawn in fine vertical lines