EMNLP Findings2025
URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models
Ruiqi Yan, Xiquan Li, Wenxi Chen, Zhikang Niu, Chen Yang, Ziyang Ma, Kai Yu, Xie Chen
Abstract
Recent advances in large language models (LLMs) have driven significant progress in endto-end spoken dialogue models (SDMs). In contrast to text-based LLMs, the evaluation framework for SDMs should encompass both cognitive dimensions (e.g., logical reasoning, knowledge) and speech-related aspects (e.g., paralinguistic cues, audio quality). However, there is still a lack of comprehensive evaluations for SDMs in speech-to-speech (S2S) scenarios. To address this gap, we propose URO-Bench, an extensive benchmark for SDMs. Notably, URO-Bench is the first S2S benchmark that covers evaluations about multilingualism, multi-round dialogues, and paralinguistics. Our benchmark is divided into two difficulty levels: basic track and pro track, each comprising 20 test sets, evaluating the spoken dialogue model's abilities in Understanding, Reasoning, and Oral conversation. Evaluations on our proposed benchmark reveal that current opensource SDMs perform rather well in daily QA tasks, but lag behind their backbone LLMs in terms of instruction-following ability and also suffer from catastrophic forgetting. Their performance in advanced evaluations of paralinguistic information and audio understanding remains subpar, highlighting the need for further research in this direction. We hope that URO-Bench can facilitate the development of spoken dialogue models by providing a multifaceted evaluation of existing models and helping to track progress in this area.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness cites Robust Speech Recognition via Large-Scale Weak Supervision
- WavLLM: Towards Robust and Adaptive Speech Large Language Model cites Robust Speech Recognition via Large-Scale Weak Supervision
- WavLLM: Towards Robust and Adaptive Speech Large Language Model cites SALMONN: Towards Generic Hearing Abilities for Large Language Models
- Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions cites Robust Speech Recognition via Large-Scale Weak Supervision
- SALMONN: Towards Generic Hearing Abilities for Large Language Models cites Robust Speech Recognition via Large-Scale Weak Supervision
Cited by top-tier papers3
- EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language ModelsLi Zhou, Lutong Yu, You Lyu, Yihang Lin et al.ICLR 2026 · 13 citations
- SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and ColloquialnessJingyu Lu, Yuhan Wang, Fan Zhuo, Xize Cheng et al.ACL 2026 · 3 citations
- Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party InterruptionsDongwook Lee, Eunwoo Song, Che Hyun Lee, Heeseung Kim et al.ACL 2026
Builds on9
- Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- SALMONN: Towards Generic Hearing Abilities for Large Language ModelsChangli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen et al.ICLR 2024 · 557 citations
- WavLLM: Towards Robust and Adaptive Speech Large Language ModelShujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen et al.EMNLP Findings 2024 · 56 citations
- MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn DialoguesGe Bai, Jie Liu, Xingyuan Bu, Yancheng He et al.ACL 2024 · 35 citations
Related papers
- Benchmarking Contextual and Paralinguistic Reasoning in Speech-LLMs: A Case Study with In-the-Wild DataQiongqiong Wang, Hardik B. Sailor, Tianchi Liu, Wenyu Zhang et al.EMNLP Findings 2025
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning BenchmarkDingdong Wang, Junan Li, Jincenzi Wu, Dongchao Yang et al.ICLR 2026 · 143 citations
- S2S-Arena: Evaluating Paralinguistic Instruction Following in Speech-to-Speech ModelsFeng Jiang, Zhiyu Lin, Yiyang Liu, Liumeng Xue et al.ACL 2026 · 17 citations
- OMHBench: Benchmarking Balanced and Grounded Omni-Modal Multi-Hop ReasoningSeunghee Kim, Ingyu Bang, Seokgyu Jang, Changhyeon Kim et al.ACL Findings 2026
- Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language ModelsKuofeng Gao, Shutao Xia, Ke Xu, Philip Torr et al.ACL 2025
