Lune

ACL Findings2023

UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning

Heqing Zou, Meng Shen, Chen Chen, Yuchen Hu, Deepu Rajan, Eng Siong Chng

Abstract

Multimodal learning aims to imitate human beings to acquire complementary information from multiple modalities for various downstream tasks. However, traditional aggregationbased multimodal fusion methods ignore the inter-modality relationship, treat each modality equally, suffer sensor noise, and thus reduce multimodal learning performance. In this work, we propose a novel multimodal contrastive method to explore more reliable multimodal representations under the weak supervision of unimodal predicting. Specifically, we first capture task-related unimodal representations and the unimodal predictions from the introduced unimodal predicting task. Then the unimodal representations are aligned with the more effective one by the designed multimodal contrastive method under the supervision of the unimodal predictions. Experimental results with fused features on two image-text classification benchmarks UPMC-Food-101 and N24News show that our proposed Unimodality-Supervised MultiModal Contrastive (UniS-MMC) learning method outperforms current state-of-the-art multimodal methods. The detailed ablation study and analysis further demonstrate the advantage of our proposed method.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • IBMA: Information Bottleneck-Based Multimodal Alignment cites Learning Transferable Visual Models From Natural Language Supervision
  • Semi-Supervised Multimodal Classification Through Learning from Modal and Strategic Complementarities cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
  • Align before Fuse: Vision and Language Representation Learning with Momentum Distillation cites Learning Transferable Visual Models From Natural Language Supervision
  • C3Net: Compound Conditioned ControlNet for Multimodal Content Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
  • ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. C3Net: Compound Conditioned ControlNet for Multimodal Content GenerationJuntao Zhang, Yuehuai Liu, Yu-Wing Tai, Chi-Keung TangCVPR 2024 · 4 citations
  2. Semi-Supervised Multimodal Classification Through Learning from Modal and Strategic ComplementaritiesJunchi Chen, Richong Zhang, Junfan ChenAAAI 2025 · 1 citation
  3. IBMA: Information Bottleneck-Based Multimodal AlignmentYancheng Wang, Zeyu Dong, Dongfang Sun, Alvin Silva et al.ICML 2026
  4. BayesVQA: Energy-Guided Bayesian Debiasing for Language-Bias-Robust Visual Question AnsweringZhiqi Huang, Huanjia Zhu, Xiangwen Deng, Qinghao Zhong et al.AAAI 2026

Builds on20

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
  4. Align before Fuse: Vision and Language Representation Learning with Momentum DistillationJunnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafiq R. Joty et al.NeurIPS 2021 · 2,985 citations
  5. ViLT: Vision-and-Language Transformer Without Convolution or Region SupervisionWonjae Kim, Bokyung Son, Ildoo KimICML 2021 · 2,258 citations

Related papers

  1. Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment AnalysisRonghao Lin, Haifeng HuEMNLP Findings 2022 · 27 citations
  2. Improving Multimodal Sentiment Analysis: Supervised Angular margin-based Contrastive Learning for Enhanced Fusion RepresentationCong-Duy Nguyen, Thong Nguyen, Duc Anh Vu, Anh Tuan LuuEMNLP Findings 2023 · 11 citations
  3. UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive LearningWei Li, Can Gao, Guocheng Niu, Xinyan Xiao et al.ACL 2021
  4. Multimedia Event Extraction From News With a Unified Contrastive Learning FrameworkJian Liu, Yufeng Chen, Jinan XuACM MM 2022 · 12 citations
  5. On Uni-Modal Feature Learning in Supervised Multi-Modal LearningChenzhuang Du, Jiaye Teng, Tingle Li, Yichen Liu et al.ICML 2023 · 79 citations
Dusk over the sea between two cliffs drawn in fine vertical lines