Beyond Accuracy: An Empirical Study of Perception Stability in Multimodal Large Language Models

Recent multimodal large language models (MLLMs) are predominantly evaluated on aggregate accuracy across perception benchmarks, with far less attention to how stable their performance is across tasks, domains, and evaluation factors. This accuracy-centric focus yields a limited and sometimes misleading picture of multimodal perception capability. In this paper, we take a step toward systematically studying perception stability and analyzing how it scales across pre-training, post-training, and inference. We begin by decomposing 11 perception benchmarks into 6 core abilities and evaluating 18 MLLMs with two stability metrics that measure ranking volatility and performance dispersion across heterogeneous subtasks. Our findings show that perception stability is a fundamental yet long-overlooked dimension of MLLM behavior. Specifically: (i) although recent open-source MLLMs can match or surpass strong closed-source models in accuracy, they exhibit a substantial gap in perception stability; (ii) training dynamics reveal an ability conflict phenomenon, where gains in one or more abilities coincide with significant declines in another ability's accuracy and stability; and (iii) stability is shaped by different factors at different stages: during pre-training and post-training it is primarily driven by the data mixing ratio and LLM size, whereas at inference time it is strongly influenced by the model's reasoning style. Finally, targeted fine-tuning and model merging partially mitigate ability conflicts, but fully resolving multiple concurrent conflicts remains an open problem. Our code is available at https: //github.com/Chenfeng1271/AbilityLens.