ACL Findings2025
On the Robust Approximation of ASR Metrics
Abstract
Recent advances in speech foundation models are largely driven by scaling both model size and data, enabling them to perform a wide range of tasks, including speech recognition.Traditionally, ASR models are evaluated using metrics like Word Error Rate (WER) and Character Error Rate (CER), which depend on ground truth labels.As a result of limited labeled data from diverse domains and testing conditions, the true generalization capabilities of these models beyond standard benchmarks remain unclear.Moreover, labeling data is both costly and time-consuming.To address this, we propose a novel label-free approach for approximating ASR performance metrics, eliminating the need for ground truth labels.Our method utilizes multimodal embeddings in a unified space for speech and transcription representations, combined with a high-quality proxy model to compute proxy metrics.These features are used to train a regression model to predict key ASR metrics like Word Error Rate (WER) and Character Error Rate (CER).We experiment with over 40 models across 14 datasets representing both standard and in-thewild testing conditions.Our results show that we approximate the metrics within a singledigit absolute difference across all experimental configurations, outperforming the most recent baseline by more than 50%.MD Rio, Peter Ha, Quinten McNamara, Corey Miller, and Shipra Chandra.2022.Earnings-22: A practical benchmark for accents in the wild.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
- Robust Speech Recognition via Large-Scale Weak Supervision cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing cites wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Builds on7
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech RepresentationsAlexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, Michael AuliNeurIPS 2020 · 9,451 citations
- Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
- data2vec: A General Framework for Self-supervised Learning in Speech, Vision and LanguageAlexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu et al.ICML 2022 · 1,123 citations
- Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text TasksYun Tang, Anna Y. Sun, Hirofumi Inaguma, Xinyue Chen et al.ACL 2023 · 8 citations
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language ProcessingJunyi Ao, Rui Wang, Long Zhou, Chengyi Wang et al.ACL 2022
Related papers
- Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation ModelsHanin Atwany, Abdul Waheed, Rita Singh, Monojit Choudhury et al.ACL Findings 2025 · 6 citations
- BLASER: A Text-Free Speech-to-Speech Translation Evaluation MetricMingda Chen, Paul-Ambroise Duquenne, Pierre Andrews, Justine Kao et al.ACL 2023 · 8 citations
- Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation ModelsYuchen Hu, Chen Chen, Chao-Han Huck Yang, Chengwei Qin et al.NeurIPS 2024 · 14 citations
- MT-Ranker: Reference-free machine translation evaluation by inter-system rankingIbraheem Muhammad Moosa, Rui Zhang, Wenpeng YinICLR 2024 · 13 citations
- Speech Robust Bench: A Robustness Benchmark For Speech RecognitionMuhammad A. Shah, David Solans Noguero, Mikko A. Heikkilä, Bhiksha Raj et al.ICLR 2025
