Lune

ACL Findings2024

wav2vec-S: Adapting Pre-trained Speech Models for Streaming

Biao Fu, Kai Fan, Minpeng Liao, Yidong Chen, Xiaodong Shi, Zhongqiang Huang

Abstract

Pre-trained speech models, such as wav2vec 2.0, have significantly advanced speech-related tasks, including speech recognition and translation. However, their applicability in streaming scenarios is limited because these models are trained on complete utterances, leading to a mismatch with incremental streaming inputs. This paper identifies three critical design aspects within the architecture of wav2vec 2.0 and proposes a novel model, wav2vec-S, which incorporates simple modifications to ensure consistent speech representations during both training and inference phases for streaming speech inputs. Furthermore, we demonstrate that wav2vec-S models can be efficiently adapted from pre-trained wav2vec 2.0 models through continued pre-training and effectively finetuned to meet various latency requirements in downstream applications. Experiments on speech recognition and translation tasks show that wav2vec-S outperforms strong baseline models and achieves a superior balance between quality and latency.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • BEATs: Audio Pre-Training with Acoustic Tokenizers cites Masked Autoencoders that Listen
  • BEATs: Audio Pre-Training with Acoustic Tokenizers cites SSAST: Self-Supervised Audio Spectrogram Transformer
  • Masked Autoencoders that Listen cites SSAST: Self-Supervised Audio Spectrogram Transformer
  • Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional Architecture cites Robust Speech Recognition via Large-Scale Weak Supervision
  • Phonotomizer: A Compact, Unsupervised, Online Training Approach to Real-Time, Multilingual Phonetic Segmentation cites Robust Speech Recognition via Large-Scale Weak Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Efficient and Adaptive Simultaneous Speech Translation with Fully Unidirectional ArchitectureBiao Fu, Donglei Yu, Minpeng Liao, Chengxi Li et al.AAAI 2026 · 1 citation
  2. Phonotomizer: A Compact, Unsupervised, Online Training Approach to Real-Time, Multilingual Phonetic SegmentationMichael S. Yantosca, Albert M. K. ChengACL 2025

Builds on25

  1. Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
  2. BEATs: Audio Pre-Training with Acoustic TokenizersSanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu et al.ICML 2023 · 568 citations
  3. Masked Autoencoders that ListenPo-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski et al.NeurIPS 2022 · 524 citations
  4. SSAST: Self-Supervised Audio Spectrogram TransformerYuan Gong, Cheng-I Lai, Yu-An Chung, James R. GlassAAAI 2022 · 397 citations
  5. Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and LanguageAlexei Baevski, Arun Babu, Wei-Ning Hsu, Michael AuliICML 2023 · 137 citations

Related papers

  1. RedApt: An Adaptor for wav2vec 2 EncodingFaster and Smaller Speech Translation without Quality CompromiseJinming Zhao, Hao Yang, Gholamreza Haffari, Ehsan ShareghiEMNLP Findings 2022
  2. Self-supervised learning with random-projection quantizer for speech recognitionChung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu et al.ICML 2022 · 245 citations
  3. Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and InferenceBiao Fu, Minpeng Liao, Kai Fan, Zhongqiang Huang et al.EMNLP 2023 · 4 citations
  4. SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech TranslationKeqi Deng, Wenxi Chen, Xie Chen, Philip C. WoodlandACL 2025
  5. Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech RecognitionGuolin Zheng, Yubei Xiao, Ke Gong, Pan Zhou et al.EMNLP Findings 2021 · 19 citations
Dusk over the sea between two cliffs drawn in fine vertical lines