Lune

EMNLP Findings2025

Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness

Yusheng Zhao, Xiao Luo, Junyu Luo, Weizhi Zhang, Zhiping Xiao, Wei Ju, Philip S. Yu, Ming Zhang

Abstract

Multi-modal large language models (MLLMs) have recently achieved great success in processing and understanding information from diverse modalities (e.g., text, audio, and visual signals). Despite their growing popularity, there remains a lack of comprehensive evaluation measuring the audio-visual capabilities of these models, especially in diverse scenarios (e.g., distribution shifts and adversarial attacks). In this paper, we present a multifaceted evaluation of the audio-visual capability of MLLMs, focusing on four key dimensions: effectiveness, efficiency, generalizability, and robustness. Through extensive experiments, we find that MLLMs exhibit strong zero-shot and few-shot generalization abilities, enabling them to achieve great performance with limited data. However, their success relies heavily on the vision modality, which impairs performance when visual input is corrupted or missing. Additionally, while MLLMs are susceptible to adversarial samples, they demonstrate greater robustness compared to traditional models. The experimental results and our observations provide new insights into the audio-visual capabilities of MLLMs, highlighting areas for improvement and offering guidance for future research.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Efficient Test-Time Model Adaptation without Forgetting cites Tent: Fully Test-Time Adaptation by Entropy Minimization
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. MMClima: A Framework for Multimodal Climate Science Data and EvaluationMuhammad Umer Sheikh, Hassan Abid, Khawar shehzad, Ufaq Khan et al.ICML 2026 · 2 citations
  2. Sample Lottery: Unsupervised Discovery of Critical Instances for LLM ReasoningZhiping Xiao, Yusheng Zhao, Qixin Zhang, Jiaye Xie et al.ICLR 2026

Builds on31

  1. ViViT: A Video Vision TransformerAnurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun et al.ICCV 2021 · 2,947 citations
  2. Tent: Fully Test-Time Adaptation by Entropy MinimizationDequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen et al.ICLR 2021 · 1,731 citations
  3. Are We on the Right Way for Evaluating Large Vision-Language Models?Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang et al.NeurIPS 2024 · 1,029 citations
  4. NExT-GPT: Any-to-Any Multimodal LLMShengqiong Wu, Hao Fei, Leigang Qu, Wei Ji et al.ICML 2024 · 786 citations
  5. Efficient Test-Time Model Adaptation without ForgettingShuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen et al.ICML 2022 · 579 citations

Related papers

  1. MISSRAG: Addressing the Missing Modality Challenge in Multimodal Large Language ModelsVittorio Pipoli, Alessia Saporita, Federico Bolelli, Marcella Cornia et al.ICCV 2025 · 4 citations
  2. AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMsSanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Yaoting Wang et al.ICCV 2025 · 1 citation
  3. AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language ModelsSung-Bin Kim, Oh Hyun-Bin, JungMok Lee, Arda Senocak et al.ICLR 2025
  4. Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?Hengyi Feng, Zeang Sheng, Meiyi Qiang, Yang Li et al.ACL Findings 2026
  5. Benchmarking Multimodal Large Language Models Against Image CorruptionsXinkuan Qiu, Meina Kan, Yongbin Zhou, Shiguang ShanICCV 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines