Lune

MobiCom2024顶会

Turbocharge Speech Understanding with Pilot Inference

Rongxiang Wang, Felix Xiaozhu Lin

摘要

Modern speech understanding (SU) runs a sophisticated pipeline: ingesting streaming voice input, the pipeline executes encoder-decoder based deep neural networks repeatedly; by doing so, the pipeline generates tentative outputs (called hypotheses), and periodically scores the hypotheses.

This paper sets to accelerate SU on resource-constrained edge devices. It takes a hybrid approach: to speed up ondevice execution; to offload inputs that are beyond the device's capacity. While the approach is well-known, we address SU's unique challenges with novel techniques: (1) late contextualization, which executes a model's attentive encoder in parallel to the input ingestion; (2) pilot inference, which mitigates the SU pipeline's temporal load imbalance; (3) autoregression offramps, which evaluate offloading decisions based on pilot inferences and hypotheses.

Our techniques are compatible with existing speech models, pipelines, and frameworks; they can be applied independently or in combination. Our prototype, called PASU, is tested on Arm platforms with 6 -8 cores: it delivers SOTA accuracy; it reduces the end-to-end latency by 2x and reduces the offloading needs by 2x.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 1 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • Robust Speech Recognition via Large-Scale Weak Supervision 引用了 wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
  • SILENCE: Protecting privacy in offloaded speech understanding on resource-constrained devices 引用了 Robust Speech Recognition via Large-Scale Weak Supervision
  • SILENCE: Protecting privacy in offloaded speech understanding on resource-constrained devices 引用了 wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper1

  1. SILENCE: Protecting privacy in offloaded speech understanding on resource-constrained devicesDongqi Cai, Shangguang Wang, Zeling Zhang, Felix Xiaozhu Lin 等NeurIPS 2024 · 2 次引用

它引用的 Paper12

  1. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 次引用
  2. Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman 等ICML 2023 · 6,966 次引用
  3. BERT Loses Patience: Fast and Robust Inference with Early ExitWangchunshu Zhou, Canwen Xu, Tao Ge, Julian J. McAuley 等NeurIPS 2020 · 473 次引用
  4. Speculative Decoding with Big Little DecoderSehoon Kim, Karttikeya Mangalam, Suhong Moon, Jitendra Malik 等NeurIPS 2023 · 212 次引用
  5. Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and UnderstandingYifan Peng, Siddharth Dalmia, Ian R. Lane, Shinji WatanabeICML 2022 · 203 次引用

相关 Paper

  1. Rethinking Pruning for Accelerating Deep Inference At the EdgeDawei Gao, Xiaoxi He, Zimu Zhou, Yongxin Tong 等KDD 2020 · 24 次引用
  2. Speculative Speculative DecodingTanishq Kumar, Tri Dao, Avner MayICLR 2026 · 15 次引用
  3. InfScaler: Enabling Efficient ML Inference Serving on Multi-Accelerator Edge Devices via Asymmetric Auto-ScalingBorui Li, Tiange Xia, Shuai Wang, Shuai WangDAC 2025 · 2 次引用
  4. PipeSD: An Efficient Cloud-Edge Collaborative Pipeline Inference Framework with Speculative DecodingHan Yunhe, Yunqi Gao, Bing Hu, Mahdi Boloursaz Mashhadi 等ICML 2026 · 2 次引用
  5. DIAA: A Decoding-Efficient Inference Acceleration Approach for On-Device Large Language ModelsHao Tian, Sheng Lu, Fuwen Tian, Guangming Cui 等AAAI 2026
黄昏的海面,两侧是细线勾勒的悬崖