Lune

ACL Findings2026

Stable Evidence, Unstable Decisions: An Empirical Analysis of Model Decision Stability in Vision-Language Models

Ali Khoramfar, Mohammad Javad Dousti, Alireza Mohamadian, Heshaam Faili

Abstract

VLMs provide visual information alongside their predictions, but it remains unclear whether consistency in such information implies consistent decisions. We study this question in a controlled medical-imaging setting using brain MRI with pathologyconfirmed labels and expert lesion annotations. For each human subject and modality, we construct configurations that retain the lesion content while varying surrounding context and scale and measure decision flips together with consistency in model-reported influential slices. Across four diverse VLMs (including proprietary, open-source, and domain-specific models), flip rates reach up to 75% across lesion-containing presentations, often despite high overlap in reported evidence. When lesion-related content is removed, proprietary models rarely produce a categorical diagnosis, with abstention rates ranging from 63% to 99%. These results reveal a mismatch between reported evidence and decisions, motivating evaluation beyond accuracy. Our evaluation dataset is publicly available on Hugging Face 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on2

  1. Not All Views Are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation ModelsMateusz Michalkiewicz, Sheena Bai, Mahsa Baktashmotlagh, Varun Jampani et al.ICCV 2025 · 8 citations
  2. MM-R³: On (In-)Consistency of Vision-Language Models (VLMs)Shih-Han Chou, Shivam Chandhok, Jim Little, Leonid SigalACL Findings 2025 · 3 citations

Related papers

  1. Dynamic Decision Learning: Test-Time Evolution for Abnormality Grounding in Rare DiseasesJun Li, Mingxuan Liu, Jiazhen Pan, che liu et al.ICML 2026
  2. Medic-AD: Towards Medical Vision-Language Model's Clinical IntelligenceWoohyeon Park, Jaeik Kim, Sunghwan Steve Cho, Pa Hong et al.CVPR 2026
  3. Discrepancy Ratio: Evaluating Model Performance When Even Experts Disagree on the TruthIgor Lovchinsky, Alon Daks, Israel Malkin, Pouya Samangouei et al.ICLR 2020 · 11 citations
  4. Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?Letitia Parcalabescu, Anette FrankICLR 2025
  5. Medical thinking with multiple imagesZonghai Yao, Benlu Wang, Yifan Zhang, Junda Wang et al.ICLR 2026 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines