Turbocharge Speech Understanding with Pilot Inference
Rongxiang Wang, Felix Xiaozhu Lin
摘要
Modern speech understanding (SU) runs a sophisticated pipeline: ingesting streaming voice input, the pipeline executes encoder-decoder based deep neural networks repeatedly; by doing so, the pipeline generates tentative outputs (called hypotheses), and periodically scores the hypotheses.
This paper sets to accelerate SU on resource-constrained edge devices. It takes a hybrid approach: to speed up ondevice execution; to offload inputs that are beyond the device's capacity. While the approach is well-known, we address SU's unique challenges with novel techniques: (1) late contextualization, which executes a model's attentive encoder in parallel to the input ingestion; (2) pilot inference, which mitigates the SU pipeline's temporal load imbalance; (3) autoregression offramps, which evaluate offloading decisions based on pilot inferences and hypotheses.
Our techniques are compatible with existing speech models, pipelines, and frameworks; they can be applied independently or in combination. Our prototype, called PASU, is tested on Arm platforms with 6 -8 cores: it delivers SOTA accuracy; it reduces the end-to-end latency by 2x and reduces the offloading needs by 2x.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 1 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- Robust Speech Recognition via Large-Scale Weak Supervision 引用了 wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
- SILENCE: Protecting privacy in offloaded speech understanding on resource-constrained devices 引用了 Robust Speech Recognition via Large-Scale Weak Supervision
- SILENCE: Protecting privacy in offloaded speech understanding on resource-constrained devices 引用了 wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
引用它的顶会 Paper1
它引用的 Paper12
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 次引用
- Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman 等ICML 2023 · 6,966 次引用
- BERT Loses Patience: Fast and Robust Inference with Early ExitWangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley 等NeurIPS 2020 · 473 次引用
- Speculative Decoding with Big Little DecoderSehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik 等NeurIPS 2023 · 212 次引用
- Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and UnderstandingYifan Peng, Siddharth Dalmia, Ian R. Lane, Shinji WatanabeICML 2022 · 203 次引用
相关 Paper
- Rethinking Pruning for Accelerating Deep Inference At the EdgeDawei Gao, Xiaoxi He, Zimu Zhou, Yongxin Tong 等KDD 2020 · 24 次引用
- Speculative Speculative DecodingTanishq Kumar, Tri Dao, Avner MayICLR 2026 · 15 次引用
- InfScaler: Enabling Efficient ML Inference Serving on Multi-Accelerator Edge Devices via Asymmetric Auto-ScalingBorui Li, Tiange Xia, Shuai Wang, Shuai WangDAC 2025 · 2 次引用
- PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative DecodingHan Yunhe, Yunqi Gao, Bing Hu, Mahdi Boloursaz Mashhadi 等ICML 2026 · 2 次引用
- DIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language ModelsHao Tian, Sheng Lu, Fuwen Tian, Guangming Cui 等AAAI 2026
