Lune

ACL Findings2026

Pardon? Evaluating Conversational Repair in Large Audio-Language Models

Shuanghong Huang, Jinlei Xu, Youchao Zhou, Yanghao Zhou, Xuan Zhao, Chong Feng, Wenxuan Zhang

Abstract

Large Audio-Language Models (LALMs) have demonstrated strong performance in spoken question answering (QA), with existing evaluations primarily focusing on answer accuracy and robustness to acoustic perturbations. However, such evaluations implicitly assume that spoken inputs remain semantically answerable, an assumption that often fails in real-world interaction when essential information is missing. In this work, we introduce a repair-aware evaluation setting that explicitly distinguishes between answerable and unanswerable audio inputs. We define answerability as a property of the input itself and construct paired evaluation conditions using a semantic-acoustic masking protocol. Based on this setting, we propose the Evaluability Awareness and Repair (EAR) score, a non-compensatory metric that jointly evaluates task competence under answerable conditions and repair behavior under unanswerable conditions. Experiments on two spoken QA benchmarks across diverse LALMs reveal a consistent gap between answer accuracy and conversational reliability: while many models perform well when inputs are answerable, most fail to recognize semantic unanswerability and initiate appropriate conversational repair. These findings expose a limitation of prevailing accuracy-centric evaluation practices and motivate reliability assessments that treat unanswerable inputs as cues for repair and continued interaction. The core code and dataset are publicly available at https://github.com/sheunghung/EAR.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models cites Listen, Think, and Understand
  • Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models cites Robust Speech Recognition via Large-Scale Weak Supervision
  • Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models cites SALMONN: Towards Generic Hearing Abilities for Large Language Models
  • SALMONN: Towards Generic Hearing Abilities for Large Language Models cites Listen, Think, and Understand
  • SALMONN: Towards Generic Hearing Abilities for Large Language Models cites Robust Speech Recognition via Large-Scale Weak Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on11

  1. Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, Greg Brockman et al.ICML 2023 · 6,966 citations
  2. SALMONN: Towards Generic Hearing Abilities for Large Language ModelsChangli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen et al.ICLR 2024 · 557 citations
  3. Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language ModelsSreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar et al.NeurIPS 2025 · 299 citations
  4. Listen, Think, and UnderstandYuan Gong, Hongyin Luo, Alexander H. Liu, Leonid Karlinsky et al.ICLR 2024 · 247 citations
  5. Can AI Assistants Know What They Don't Know?Qinyuan Cheng, Tianxiang Sun, Xiangyang Liu, Wenwei Zhang et al.ICML 2024 · 48 citations

Related papers

  1. Towards Reliable Large Audio Language ModelZiyang Ma, Xiquan Li, Yakun Song, Wenxi Chen et al.ACL Findings 2025
  2. Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive SurveyChih-Kai Yang, Neo S. Ho, Hung-yi LeeEMNLP 2025 · 7 citations
  3. Assistive Prompt Mediation: Evaluating Language Models Under Accessibility ConstraintsPriyaranjan Pattnayak, Ishan BanerjeeICML 2026
  4. HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to InteractionZhong Muyan, Erfei Cui, Sen Xing, Weiyun Wang et al.CVPR 2026
  5. Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language ModelsKuofeng Gao, Shutao Xia, Ke Xu, Philip Torr et al.ACL 2025
Dusk over the sea between two cliffs drawn in fine vertical lines