Lune

EMNLP Findings2023

SHARCS: Efficient Transformers Through Routing with Dynamic Width Sub-networks

Mohammadreza Salehi, Sachin Mehta, Aditya Kusupati, Ali Farhadi, Hannaneh Hajishirzi

Abstract

We introduce SHARCS for adaptive inference that takes into account the hardness of input samples. SHARCS can train a router on any transformer network, enabling the model to direct different samples to sub-networks with varying widths. Our experiments demonstrate that: (1) SHARCS outperforms or complements existing per-sample adaptive inference methods across various classification tasks in terms of accuracy vs. FLOPs; (2) SHARCS generalizes across different architectures and can be even applied to compressed and efficient transformer encoders to further improve their efficiency; (3) SHARCS can provide a 2× inference speed up at an insignificant drop in accuracy.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • TinyBERT: Distilling BERT for Natural Language Understanding cites MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
  • MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices cites TinyBERT: Distilling BERT for Natural Language Understanding
  • MatFormer: Nested Transformer for Elastic Inference cites Language Models are Few-Shot Learners
  • HydraViT: Stacking Heads for a Scalable ViT cites MatFormer: Nested Transformer for Elastic Inference
  • Movement Pruning: Adaptive Sparsity by Fine-Tuning cites TinyBERT: Distilling BERT for Natural Language Understanding
  • Movement Pruning: Adaptive Sparsity by Fine-Tuning cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. MatFormer: Nested Transformer for Elastic InferenceDevvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers et al.NeurIPS 2024 · 97 citations
  2. HydraViT: Stacking Heads for a Scalable ViTJanek Haberer, Ali Hojjat, Olaf LandsiedelNeurIPS 2024 · 11 citations
  3. SoK: Efficiency Robustness of Dynamic Deep Learning SystemsRavishka Rathnasuriya, Tingxi Li, Zexin Xu, Zihe Song et al.USENIX Security 2025

Builds on16

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  3. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
  4. MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesZhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu et al.ACL 2020 · 660 citations
  5. Movement Pruning: Adaptive Sparsity by Fine-TuningVictor Sanh, Thomas Wolf, Alexander M. RushNeurIPS 2020 · 656 citations

Related papers

  1. Adaptive Depth Networks with Skippable Sub-PathsWoochul Kang, Hyungseop LeeNeurIPS 2024 · 5 citations
  2. Adaptive Computation Modules: Granular Conditional Computation for Efficient InferenceBartosz Wójcik, Alessio Devoto, Karol Pustelnik, Pasquale Minervini et al.AAAI 2025 · 8 citations
  3. AdaPerceiver: Transformers with Adaptive Width, Depth, and TokensPurvish Jajal, Nicholas John Eliopoulos, Benjamin Shiue-Hal Chou, George K Thiruvathukal, Yung-Hsiang Lu, James C. DavisCVPR Findings 2026
  4. Faster Depth-Adaptive TransformersYijin Liu, Fandong Meng, Jie Zhou, Yufeng Chen et al.AAAI 2021 · 44 citations
  5. AdaBM: On-the-Fly Adaptive Bit Mapping for Image Super-ResolutionCheeun Hong, Kyoung Mu LeeCVPR 2024
Dusk over the sea between two cliffs drawn in fine vertical lines