HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction

Multimodal large language models (MLLMs) have expanded from vision-language systems to include audio, unlocking new capabilities in cross-modal reasoning and interaction. To address the limitation that existing benchmarks focus mainly on perception tasks and lack a unified cognitive evaluation framework, we propose Hierarchical Audio-Visual Evaluation Benchmark (HAVE-Bench). It systematically evaluates the audio-related capabilities of MLLMs along a three-level cognitive hierarchy: Perception, Reasoning, and Interaction, utilizing 2,451 curated samples and manually annotated multi-turn interaction-level tasks. Experiments using this unified framework reveal significant gaps in existing models at the reasoning and interaction levels, with speech-driven visual question answering (VQA) performance significantly lagging behind the text-image setting. These findings underscore the urgency of enhancing models' handling of long and complex audio and facilitating the transfer of reasoning capabilities from the vision-text to the audio-visual domain. Audio: [Speech] Is this an image of Mont Tendre? Answer: Yes. Audio: Sound of the oven door closing, a button press, and oven running. Question: What button on the panel did the person in the picture press? Answer: "Start" button. Audio: Recording of a math lecture. Question: What process do the two blocks in the figure demonstrate? Answer: Linear transformation. Answer: Go straight, turn right at the third intersection, then keep going straight. Audio: [Speech] Take me to the intersection of China Basin St and 4th St.