Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models

The success of vision-language models is primarily attributed to effective alignment across modalities such as vision and language. However, modality gaps persist in existing alignment algorithms and appear necessary for human perception -evident in modality-specific phenomena like visual texture and linguistic tone. These observations motivate us to computationally measure and leverage modality gaps to improve downstream tasks. We first introduce the Modality Dominance Score (MDS), which attributes multimodal features to specific modalities by categorizing them into three classes: vision-dominant features, language-dominant features, and cross-modal features. We then propose automatic interpretability metrics to evaluate these modalityspecific features in a scalable manner. Finally, we demonstrate that the training-free model editing enhances multiple downstream tasks, including mitigating bias in gender classification, generating cross-modal adversarial examples, and enabling modality-specific control in text-to-image generation. Combined with taskagnostic interpretability tools, our work offers insights for systematic analysis and lightweight editing of multimodal models. 1. We demonstrate that modality-specific information can be extracted from VLMs-specifically, text-dominant (TextD), image-dominant (ImgD), and cross-modal (CrossD) features-and show that these features exhibit distinct activation patterns when processing images versus text. 2. We propose embedding-based interpretability metrics to measure monosemanticity (withinmodality coherence) and modality fidelity (crossmodality validation) in a multimodal setup. These metrics are scalable and compatible with the existing top-k activated interpretations.