FACT: Fuzzy Alignment with Comorbidity Topology for Reliable Multi-Label Medical Image Diagnosis

In clinical practice, patients often present with multiple co-occurring diseases, yet most existing Multi-Label-Diagnosis (MLD) methods treat diagnosis as a rigid discriminative partitioning task, implicitly assuming that overlapping pathologies are separable. This assumption is problematic in medical images, where identical or highly similar visual observations may simultaneously support multiple disease labels, and disease concepts are inherently correlated rather than independent. Enforcing hard decision boundaries under such overlap suppresses shared evidence, biases feature representations, and ultimately undermines model reliability. To address this limitation, we propose Fuzzy Alignment with Comorbidity Topology (FACT), a novel paradigm that reformulates MLD as a fuzzy alignment problem between atomic visual evidence and disease semantic anchors. FACT is characterized by three key features: (1) modeling visual polysemy through shared and reusable atomic visual evidence; (2) encoding disease correlation via semantic anchors structured by comorbidity topology; and

(3) employing a metric-based fuzzy membership function for non-discriminative visual-semantic alignment. Extensive experiments on three public clinical benchmarks, together with additional evaluation under long-tailed and noisy settings, demonstrate that FACT consistently improves diagnostic performance while delivering clinically plausible predictions. The code is available at https://github.com/yyuChen9/FACT.