Towards Out-of-Modal Generalization without Instance-level Modal Correspondence
Zhuo Huang, Gang Niu, Bo Han, Masashi Sugiyama, Tongliang Liu
Abstract
The world is understood from various modalities, such as appearance, sound, and language. Since each modality only partially represents objects in a certain meaning, leveraging additional ones is beneficial in both theory and practice. However, exploiting novel modalities normally requires cross-modal pairs corresponding to the same instance, which is extremely resource-consuming and sometimes even impossible, making knowledge exploration of novel modalities largely restricted. To seek practical multi-modal learning, here we study Outof-Modal (OOM) Generalization as an initial attempt to generalize to an unknown modality without given instance-level modal correspondence. Specifically, we consider Semi-Supervised and Unsupervised scenarios of OOM Generalization, where the first has scarce correspondences and the second has none, and propose Connect&Explore (COX) to solve these problems. COX first connects OOM data and known In-Modal (IM) data through a variational information bottleneck framework to extract shared information. Then, COX leverages the shared knowledge to create emergent correspondences, which is theoretically justified from an information-theoretic perspective. As a result, the label information on OOM data emerges along with the correspondences, which helps explore the OOM data with unknown knowledge, thus benefiting generalization results. We carefully evaluate the proposed COX method under various OOM generalization scenarios, verifying its effectiveness and extensibility. The code is available at https://github.com/tmllab/2025 ICLR COX.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- TextME: Bridging Unseen Modalities Through Text Descriptions cites Learning Transferable Visual Models From Natural Language Supervision
- Continual Multimodal Contrastive Learning cites Learning Transferable Visual Models From Natural Language Supervision
- Mobile-VTON: High-Fidelity On-Device Virtual Try-On cites Learning Transferable Visual Models From Natural Language Supervision
- Learning Transferable Visual Models From Natural Language Supervision cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
- Ranked from Within: Ranking Large Multimodal Models Without Labels cites Learning Transferable Visual Models From Natural Language Supervision
- Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers5
- Continual Multimodal Contrastive LearningXiaohao Liu, Xiaobo Xia, See-Kiong Ng, Tat-Seng ChuaNeurIPS 2025 · 25 citations
- LLM Unlearning with LLM BeliefsKemou Li, Qizhou Wang, Yue Wang, Fengpeng Li et al.ICLR 2026 · 20 citations
- Mobile-VTON: High-Fidelity On-Device Virtual Try-OnZhenchen Wan, Ce Chen, Runqi Lin, Jiaxin Huang et al.CVPR 2026 · 4 citations
- TextME: Bridging Unseen Modalities Through Text DescriptionsSoyeon Hong, Jinchan Kim, Jaegook You, Seungtaek Choi et al.ICML 2026
- Ranked from Within: Ranking Large Multimodal Models Without LabelsWeijie Tu, Weijian Deng, Dylan Campbell, Yu Yao et al.ICML 2025
Builds on58
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and ConfidenceKihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang et al.NeurIPS 2020 · 5,129 citations
- The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution GeneralizationDan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath et al.ICCV 2021 · 2,294 citations
- Habitat: A Platform for Embodied AI ResearchManolis Savva, Jitendra Malik, Devi Parikh, Dhruv Batra et al.ICCV 2019 · 1,863 citations
Related papers
- Learning Unseen Modality InteractionYunhua Zhang, Hazel Doughty, Cees SnoekNeurIPS 2023 · 16 citations
- Multimodal Knowledge ExpansionZihui Xue, Sucheng Ren, Zhengqi Gao, Hang ZhaoICCV 2021 · 38 citations
- Open-Set Cross Modal Generalization via Multimodal Unified RepresentationHai Huang, Yan Xia, Shulei Wang, Hanting Wang et al.ICCV 2025
- Unpaired Image-to-Speech Synthesis With Multimodal Information BottleneckShuang Ma, Daniel McDuff, Yale SongICCV 2019 · 29 citations
- Achieving Cross Modal Generalization with Multimodal Unified RepresentationYan Xia, Hai Huang, Jieming Zhu, Zhou ZhaoNeurIPS 2023 · 84 citations
