Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing

Medical multi-modal learning requires an effective fusion capability of various heterogeneous modalities. One vital challenge is how to effectively fuse modalities when their data quality varies across different modalities and patients. For example, in the TCGA benchmark, the performance of the same modality can differ between types of cancer. Moreover, data collected at different times, locations, and with varying reagents can introduce inter-modal data quality differences (i.e., Modality Batch Effect). In response, we propose Adaptive Modality Token Re-BalanCing (AMC), a novel top-down dynamic multi-modal fusion approach. The core of AMC is to quantify the significance of each modality (Top) and then fuse them according to the modality importance (Down). Specifically, we assess the quality of each input modality and then replace uninformative tokens with inter-modal tokens accordingly. The more important a modality is, the more informative tokens are retained from that modality. The self-attention will further integrate these mixed tokens to fuse multimodal knowledge. Comprehensive experiments on both medical and general multi-modal datasets demonstrate the effectiveness and generalizability of AMC. Code is available at https://github. com/PengJieb/amc .