Lune

ACL Findings2023

Masked Audio Text Encoders are Effective Multi-Modal Rescorers

Jinglun Cai, Monica Sunkara, Xilai Li, Anshu Bhatia, Xiao Pan, Sravan Bodapati

Abstract

Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition (ASR) systems. In this work, we propose Masked Audio Text Encoder (MATE), a multi-modal masked language model rescorer which incorporates acoustic representations into the input space of MLM. We adopt contrastive learning for effectively aligning the modalities by learning shared representations. We show that using a multi-modal rescorer is beneficial for domain generalization of the ASR system when target domain data is unavailable. MATE reduces word error rate (WER) by 4%-16% on indomain, and 3%-7% on out-of-domain datasets, over the text-only baseline. Additionally, with very limited amount of training data (0.8 hours) MATE achieves a WER reduction of 8%-23% over the first-pass baseline.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations cites vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on10

  1. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
  2. Multimodal Few-Shot Learning with Frozen Language ModelsMaria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami et al.NeurIPS 2021 · 1,020 citations
  3. vq-wav2vec: Self-Supervised Learning of Discrete Speech RepresentationsAlexei Baevski, Steffen Schneider, Michael AuliICLR 2020 · 730 citations
  4. Masked Language Model ScoringJulian Salazar, Davis Liang, Toan Q. Nguyen, Katrin KirchhoffACL 2020 · 167 citations
  5. SLURP: A Spoken Language Understanding Resource PackageEmanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, Verena RieserEMNLP 2020 · 129 citations

Related papers

  1. Joint Audio/Text Training for Transformer Rescorer of Streaming Speech RecognitionSuyoun Kim, Ke Li, Lucas Kabela, Ron Huang et al.EMNLP Findings 2022 · 2 citations
  2. Contrastive Learning for Task-Independent SpeechLLM-PretrainingMaike Züfle, Jan NiehuesACL Findings 2025
  3. Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-EncoderHuiwon Jang, Jihoon Tack, Daewon Choi, Jongheon Jeong et al.NeurIPS 2023 · 9 citations
  4. LegoSLM: Connecting LLM with Speech Encoder using CTC PosteriorsRao Ma, Tongzhou Chen, Kartik Audhkhasi, Bhuvana RamabhadranEMNLP Findings 2025 · 2 citations
  5. AudioMosaic: Contrastive Masked Audio Representation LearningHanxun Huang, Qizhou Wang, Xingjun Ma, Cihang Xie et al.ICML 2026 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines