MediSee: Reasoning-Based Pixel-Level Perception in Medical Images

Despite progress in pixel-level medical image perception, existing methods remain task-specific or depend on precise prompts like bounding boxes or text. However, the need for medical knowledge limits accessibility for the general public, who are more likely to use logically reasoned oral queries than domain-specific inputs. In this paper, we introduce a novel medical vision task: Medical Reasoning Segmentation and Detection (MedSD), which aims to comprehend implicit queries about medical images and generate the corresponding segmentation mask and bounding box for the target object. To accomplish this task, we first introduce a Multi-perspective, Logic-driven Medical Reasoning Segmentation and Detection (MLMR-SD) dataset, which encompasses a substantial collection of medical entity targets along with their corresponding reasoning. Furthermore, we propose MediSee, an effective baseline model designed for MedSD. The experimental results indicate that the proposed method can effectively address MedSD with implicit colloquial queries and outperform traditional medical referring segmentation methods. The MediSee project can be found here.