Rethinking Visual Counterfactual Explanations Through Region Constraint
Bartlomiej Sobieski, Jakub Grzywaczewski, Bartlomiej Sadlej, Matthew Tivnan, Przemyslaw Biecek
Abstract
Visual counterfactual explanations (VCEs) have recently gained immense popularity as a tool for clarifying the decision-making process of image classifiers. This trend is largely motivated by what these explanations promise to deliver -indicate semantically meaningful factors that change the classifier's decision. However, we argue that current state-of-the-art approaches lack a crucial component -the region constraint -whose absence prevents from drawing explicit conclusions, and may even lead to faulty reasoning due to phenomenons like confirmation bias. To address the issue of previous methods, which modify images in a very entangled and widely dispersed manner, we propose region-constrained VCEs (RVCEs), which assume that only a predefined image region can be modified to influence the model's prediction. To effectively sample from this subclass of VCEs, we propose Region-Constrained Counterfactual Schrödinger Bridges (RCSB), an adaptation of a tractable subclass of Schrödinger Bridges to the problem of conditional inpainting, where the conditioning signal originates from the classifier of interest. In addition to setting a new state-of-the-art by a large margin, we extend RCSB to allow for exact counterfactual reasoning, where the predefined region contains only the factor of interest, and incorporating the user to actively interact with the RVCE by predefining the regions manually.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Auditing Sybil: Explaining Deep Lung Cancer Risk Prediction Through Generative Interventional Attributions cites System-Embedded Diffusion Bridge Models
- Auditing Sybil: Explaining Deep Lung Cancer Risk Prediction Through Generative Interventional Attributions cites Denoising Diffusion Probabilistic Models
- Auditing Sybil: Explaining Deep Lung Cancer Risk Prediction Through Generative Interventional Attributions cites Counterfactual Explanations on Robust Perceptual Geodesics
- Segment Anything cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Segment Anything cites Learning Transferable Visual Models From Natural Language Supervision
- System-Embedded Diffusion Bridge Models cites Denoising Diffusion Probabilistic Models
- MaskDiME: Adaptive Masked Diffusion for Precise and Efficient Visual Counterfactual Explanations cites Denoising Diffusion Probabilistic Models
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites Learning Transferable Visual Models From Natural Language Supervision
- V-CECE: Visual Counterfactual Explanations via Conceptual Edits cites Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
- V-CECE: Visual Counterfactual Explanations via Conceptual Edits cites Denoising Diffusion Probabilistic Models
- Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Cited by top-tier papers6
- System-Embedded Diffusion Bridge ModelsBartlomiej Sobieski, Matthew Tivnan, Yuang Wang, Siyeop Yoon et al.NeurIPS 2025 · 4 citations
- V-CECE: Visual Counterfactual Explanations via Conceptual EditsNikolaos Spanos, Maria Lymperaiou, Giorgos Filandrianos, Konstantinos Thomas et al.NeurIPS 2025 · 4 citations
- Auditing Sybil: Explaining Deep Lung Cancer Risk Prediction Through Generative Interventional AttributionsBartlomiej Sobieski, Jakub Grzywaczewski, Karol Dobiczek, Mateusz Wójcik et al.ICML 2026 · 2 citations
- MaskDiME: Adaptive Masked Diffusion for Precise and Efficient Visual Counterfactual ExplanationsChanglu Guo, Anders Nymark Christensen, Anders Bjorholm Dahl, Morten Rieger HannemoseCVPR 2026 · 2 citations
- Counterfactual Explanations on Robust Perceptual GeodesicsEslam Zaher, Dr Maciej Trzaskowski, Quan Nguyen, Fred RoostaICLR 2026 · 2 citations
Builds on26
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
- Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao et al.ICCV 2023 · 13,211 citations
Related papers
- Diffusion Visual Counterfactual ExplanationsMaximilian Augustin, Valentyn Boreiko, Francesco Croce, Matthias HeinNeurIPS 2022 · 124 citations
- Looking in the Mirror: A Faithful Counterfactual Explanation Method for Interpreting Deep Image Classification ModelsTownim Faisal Chowdhury, Vu Minh Hieu Phan, Kewen Liao, Nanyu Dong et al.ICCV 2025
- Explanation by Progressive ExaggerationSumedha Singla, Brian Pollack, Junxiang Chen, Kayhan BatmanghelichICLR 2020 · 116 citations
- Image Safeguarding: Reasoning with Conditional Vision Language Model and Obfuscating Unsafe Content CounterfactuallyMazal Bethany, Brandon Wherry, Nishant Vishwamitra, Peyman NajafiradAAAI 2024 · 9 citations
- VeriX: Towards Verified Explainability of Deep Neural NetworksMin Wu, Haoze Wu, Clark W. BarrettNeurIPS 2023 · 39 citations
