Lune

ACL Findings2025

On the Robust Approximation of ASR Metrics

Abdul Waheed, Hanin Atwany, Rita Singh, Bhiksha Raj

Abstract

Recent advances in speech foundation models are largely driven by scaling both model size and data, enabling them to perform a wide range of tasks, including speech recognition.Traditionally, ASR models are evaluated using metrics like Word Error Rate (WER) and Character Error Rate (CER), which depend on ground truth labels.As a result of limited labeled data from diverse domains and testing conditions, the true generalization capabilities of these models beyond standard benchmarks remain unclear.Moreover, labeling data is both costly and time-consuming.To address this, we propose a novel label-free approach for approximating ASR performance metrics, eliminating the need for ground truth labels.Our method utilizes multimodal embeddings in a unified space for speech and transcription representations, combined with a high-quality proxy model to compute proxy metrics.These features are used to train a regression model to predict key ASR metrics like Word Error Rate (WER) and Character Error Rate (CER).We experiment with over 40 models across 14 datasets representing both standard and in-thewild testing conditions.Our results show that we approximate the metrics within a singledigit absolute difference across all experimental configurations, outperforming the most recent baseline by more than 50%.MD Rio, Peter Ha, Quinten McNamara, Corey Miller, and Shipra Chandra.2022.Earnings-22: A practical benchmark for accents in the wild.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
  • Robust Speech Recognition via Large-Scale Weak Supervision cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
  • SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on7

  1. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
  2. Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
  3. data2vec: A General Framework for Self-supervised Learning in Speech, Vision and LanguageAlexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu et al.ICML 2022 · 1,123 citations
  4. Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text TasksYun Tang, Anna Y. Sun, Hirofumi Inaguma, Xinyue Chen et al.ACL 2023 · 8 citations
  5. SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language ProcessingJunyi Ao, Rui Wang, Long Zhou, Chengyi Wang et al.ACL 2022

Related papers

  1. Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation ModelsHanin Atwany, Abdul Waheed, Rita Singh, Monojit Choudhury et al.ACL Findings 2025 · 6 citations
  2. BLASER: A Text-Free Speech-to-Speech Translation Evaluation MetricMingda Chen, Paul-Ambroise Duquenne, Pierre Andrews, Justine Kao et al.ACL 2023 · 8 citations
  3. Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation ModelsYuchen Hu, Chen Chen, Chao-Han Huck Yang, Chengwei Qin et al.NeurIPS 2024 · 14 citations
  4. MT-Ranker: Reference-free machine translation evaluation by inter-system rankingIbraheem Muhammad Moosa, Rui Zhang, Wenpeng YinICLR 2024 · 13 citations
  5. Speech Robust Bench: A Robustness Benchmark For Speech RecognitionMuhammad A. Shah, David Solans Noguero, Mikko A. Heikkilä, Bhiksha Raj et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines