ACL Findings2026
Afrispeech Semantics: Evaluating Audio-Semantic Reasoning in Spoken Language Models Across Domains and Accents
Abstract
Audio language models (ALMs) are increasingly used for speech-based understanding; yet, their ability to perform semantic reasoning beyond transcription, Text-to-Audio Retrieval, Captioning, and Question-Answering accuracy remains insufficiently benchmarked. In particular, the effects of accent variation, domain shift, and semantic over-inference on audio reasoning are poorly understood. We evaluate audio language models across five semantic and paralinguistic reasoning tasks: entailment, consistency, plausibility, accent drift, and accent restraint. Collectively, these tasks assess a model's ability to reason over spoken audio as the primary evidence source, including whether a textual hypothesis can be inferred, contradicted, or left undetermined by the audio, whether statements align or conflict with spoken content, whether claims are plausible given the discourse, and whether model predictions remain stable or appropriately constrained across accent variation. These findings highlight critical limitations in current audio reasoning evaluations and hope to provide guidance for more robust and equitable ALM design and assessment.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Audio Entailment: Assessing Deductive Reasoning for Audio Understanding cites GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities
- GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities cites Audio Entailment: Assessing Deductive Reasoning for Audio Understanding
- Audio-Aware Large Language Models as Judges for Speaking Styles cites SALMONN: Towards Generic Hearing Abilities for Large Language Models
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models cites Audio Entailment: Assessing Deductive Reasoning for Audio Understanding
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models cites SALMONN: Towards Generic Hearing Abilities for Large Language Models
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models cites GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities
- GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities cites SALMONN: Towards Generic Hearing Abilities for Large Language Models
Builds on8
- SALMONN: Towards Generic Hearing Abilities for Large Language ModelsChangli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen et al.ICLR 2024 · 557 citations
- Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language ModelsSreyan Ghosh, Arushi Goel, Jaehyeon Kim, Sonal Kumar et al.NeurIPS 2025 · 299 citations
- GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning AbilitiesSreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru et al.EMNLP 2024 · 34 citations
- Audio Entailment: Assessing Deductive Reasoning for Audio UnderstandingSoham Deshmukh, Shuo Han, Hazim T. Bukhari, Benjamin Elizalde et al.AAAI 2025 · 23 citations
- Audio-Aware Large Language Models as Judges for Speaking StylesCheng-Han Chiang, Xiaofei Wang, Chung-Ching Lin, Kevin Lin et al.EMNLP Findings 2025 · 4 citations
Related papers
- Audio-Reasoner: Improving Reasoning Capability in Large Audio Language ModelsZhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu et al.EMNLP 2025 · 5 citations
- AudSemThinker: Enhancing Audio-Language Models Through Reasoning over Semantics of SoundGijs Wijngaard, Elia Formisano, Michele Esposito, Michel DumontierNeurIPS 2025 · 22 citations
- UALM: Unified Audio Language Model for Understanding, Generation and ReasoningJinchuan Tian, Sang-gil Lee, Zhifeng Kong, Sreyan Ghosh et al.ICLR 2026 · 17 citations
- Mellow: a small audio language model for reasoningSoham Deshmukh, Satvik Dixit, Rita Singh, Bhiksha RajNeurIPS 2025 · 39 citations
- CompA: Addressing the Gap in Compositional Reasoning in Audio-Language ModelsSreyan Ghosh, Ashish Seth, Sonal Kumar, Utkarsh Tyagi et al.ICLR 2024 · 53 citations
