Lune

CVPR Findings2026

Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning

Mohamed Harmanani, Bining Long, Zhuoxin Guo, Paul F.R. Wilson, Amirhossein Sabour, Minh Nguyen Nhat To, Gabor Fichtinger, Purang Abolmaesumi, Parvin Mousavi

Abstract

Concept Bottleneck Models (CBMs) are a prominent framework for interpretable AI that map learned visual features onto a set of meaningful concepts, to be used for taskspecific downstream predictions. Their sequential structure enhances transparency by connecting model predictions to the underlying concepts that support them. In medical imaging, where transparency is essential, CBMs offer an appealing foundation for explainable model design. However, their discrete concept representations overlook broader clinical context such as diagnostic guidelines and expert heuristics, reducing reliability in complex cases. We propose MedCBR, a concept-based reasoning framework that integrates clinical guidelines with vision-language and reasoning models. Labeled clinical descriptors are transformed into guideline-conformant text, and a concept-based model is trained with a multi-task objective combining multi-modal contrastive alignment, concept supervision, and diagnostic classification to jointly ground image features, concepts, and pathology. A reasoning model then converts these predictions into structured clinical narratives that explain the diagnosis, emulating expert reasoning based on established guidelines. Med-CBR achieves superior diagnostic and concept-level performance, with AUROCs of 94.2% on ultrasound and 84.0% on mammography. Further experiments were also performed on non-medical datasets, with 86.1% accuracy. Our framework enhances interpretability and forms an end-to-end bridge from medical image analysis to decision-making.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Post-hoc Concept Bottleneck Models cites Concept Bottleneck Models
  • Post-hoc Concept Bottleneck Models cites Learning Transferable Visual Models From Natural Language Supervision
  • Vision Transformers Need Registers cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Concise and Descriptive Attributes for Visual Recognition cites Post-hoc Concept Bottleneck Models
  • Learning Concise and Descriptive Attributes for Visual Recognition cites Concept Bottleneck Models
  • Learning Concise and Descriptive Attributes for Visual Recognition cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on12

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Concept Bottleneck ModelsPang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann et al.ICML 2020 · 1,233 citations
  3. Vision Transformers Need RegistersTimothée Darcet, Maxime Oquab, Julien Mairal, Piotr BojanowskiICLR 2024 · 769 citations
  4. Learning Concise and Descriptive Attributes for Visual RecognitionAn Yan, Yu Wang, Yiwu Zhong, Chengyu Dong et al.ICCV 2023 · 94 citations
  5. Post-hoc Concept Bottleneck ModelsMert Yüksekgönül, Maggie Wang, James ZouICLR 2023 · 37 citations

Related papers

  1. MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept AlignmentYequan Bie, Luyang Luo, Hao ChenAAAI 2024 · 28 citations
  2. Interpretable Concept-Based Memory ReasoningDavid Debot, Pietro Barbiero, Francesco Giannini, Gabriele Ciravegna et al.NeurIPS 2024 · 26 citations
  3. Rounded or Streamlined Head? Bridging Concept Bottleneck Models and Attribute-Described Object PartsYang Liu, Jiajin Zhang, Yaojun Hu, Bingguang Hao et al.CVPR 2026
  4. Cross-Modal Conceptualization in Bottleneck ModelsDanis Alukaev, Semen Kiselev, Ilya Pershin, Bulat Ibragimov et al.EMNLP 2023 · 4 citations
  5. Object-Centric Concept-BottlenecksDavid Steinmann, Wolfgang Stammer, Antonia Wüst, Kristian KerstingNeurIPS 2025 · 12 citations
Dusk over the sea between two cliffs drawn in fine vertical lines