CVPR Findings2026
Vision-Language Models Encode Clinical Guidelines for Concept-Based Medical Reasoning
Mohamed Harmanani, Bining Long, Zhuoxin Guo, Paul F.R. Wilson, Amirhossein Sabour, Minh Nguyen Nhat To, Gabor Fichtinger, Purang Abolmaesumi, Parvin Mousavi
Abstract
Concept Bottleneck Models (CBMs) are a prominent framework for interpretable AI that map learned visual features onto a set of meaningful concepts, to be used for taskspecific downstream predictions. Their sequential structure enhances transparency by connecting model predictions to the underlying concepts that support them. In medical imaging, where transparency is essential, CBMs offer an appealing foundation for explainable model design. However, their discrete concept representations overlook broader clinical context such as diagnostic guidelines and expert heuristics, reducing reliability in complex cases. We propose MedCBR, a concept-based reasoning framework that integrates clinical guidelines with vision-language and reasoning models. Labeled clinical descriptors are transformed into guideline-conformant text, and a concept-based model is trained with a multi-task objective combining multi-modal contrastive alignment, concept supervision, and diagnostic classification to jointly ground image features, concepts, and pathology. A reasoning model then converts these predictions into structured clinical narratives that explain the diagnosis, emulating expert reasoning based on established guidelines. Med-CBR achieves superior diagnostic and concept-level performance, with AUROCs of 94.2% on ultrasound and 84.0% on mammography. Further experiments were also performed on non-medical datasets, with 86.1% accuracy. Our framework enhances interpretability and forms an end-to-end bridge from medical image analysis to decision-making.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Post-hoc Concept Bottleneck Models cites Concept Bottleneck Models
- Post-hoc Concept Bottleneck Models cites Learning Transferable Visual Models From Natural Language Supervision
- Vision Transformers Need Registers cites Learning Transferable Visual Models From Natural Language Supervision
- Learning Concise and Descriptive Attributes for Visual Recognition cites Post-hoc Concept Bottleneck Models
- Learning Concise and Descriptive Attributes for Visual Recognition cites Concept Bottleneck Models
- Learning Concise and Descriptive Attributes for Visual Recognition cites Learning Transferable Visual Models From Natural Language Supervision
Builds on12
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Concept Bottleneck ModelsPang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann et al.ICML 2020 · 1,233 citations
- Vision Transformers Need RegistersTimothée Darcet, Maxime Oquab, Julien Mairal, Piotr BojanowskiICLR 2024 · 769 citations
- Learning Concise and Descriptive Attributes for Visual RecognitionAn Yan, Yu Wang, Yiwu Zhong, Chengyu Dong et al.ICCV 2023 · 94 citations
- Post-hoc Concept Bottleneck ModelsMert Yüksekgönül, Maggie Wang, James ZouICLR 2023 · 37 citations
Related papers
- MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept AlignmentYequan Bie, Luyang Luo, Hao ChenAAAI 2024 · 28 citations
- Interpretable Concept-Based Memory ReasoningDavid Debot, Pietro Barbiero, Francesco Giannini, Gabriele Ciravegna et al.NeurIPS 2024 · 26 citations
- Rounded or Streamlined Head? Bridging Concept Bottleneck Models and Attribute-Described Object PartsYang Liu, Jiajin Zhang, Yaojun Hu, Bingguang Hao et al.CVPR 2026
- Cross-Modal Conceptualization in Bottleneck ModelsDanis Alukaev, Semen Kiselev, Ilya Pershin, Bulat Ibragimov et al.EMNLP 2023 · 4 citations
- Object-Centric Concept-BottlenecksDavid Steinmann, Wolfgang Stammer, Antonia Wüst, Kristian KerstingNeurIPS 2025 · 12 citations
