Lune

ACL Findings2023

Deeply Coupled Cross-Modal Prompt Learning

Xuejing Liu, Wei Tang, Jinghui Lu, Rui Zhao, Zhaojun Guo, Fei Tan

Abstract

Recent advancements in multimodal foundation models (e.g., CLIP) have excelled in zeroshot generalization. Prompt tuning involved in the knowledge transfer from foundation models to downstream tasks has gained significant attention recently. Existing prompttuning methods in cross-modal learning, however, either solely focus on language branch, or learn vision-language interaction in a shallow mechanism. In this context, we propose a Deeply coupled Cross-modal Prompt learning (DCP) method based on CLIP. DCP flexibly accommodates the interplay between vision and language with a Cross-Modal Prompt Attention (CMPA) mechanism, which enables the mutual exchange of respective representation through a well-connected multi-head attention module progressively and strongly. We then conduct comprehensive few-shot learning experiments on 11 image classification datasets and analyze the robustness to domain shift as well. Thorough experimental analysis evidently demonstrates the superb few-shot generalization and compelling domain adaption capacity of a well-executed DCP. The code can be found at https://github.com/GingL/CMPA.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Advancing Prompt Learning through an External Layer cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • Advancing Prompt Learning through an External Layer cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
  • Advancing Prompt Learning through an External Layer cites Learning Transferable Visual Models From Natural Language Supervision
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
  • Adaptive Prompt-Based Semantic Embedding with Inspire Potential of Implicit Knowledge for Cross-Modal Retrieval cites Language Models are Few-Shot Learners
  • Adaptive Prompt-Based Semantic Embedding with Inspire Potential of Implicit Knowledge for Cross-Modal Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
  • Link-Context Learning for Multimodal LLMs cites Language Models are Few-Shot Learners
  • Link-Context Learning for Multimodal LLMs cites Learning Transferable Visual Models From Natural Language Supervision
  • Low-rank Prompt Interaction for Continual Vision-Language Retrieval cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Low-rank Prompt Interaction for Continual Vision-Language Retrieval cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • Low-rank Prompt Interaction for Continual Vision-Language Retrieval cites Language Models are Few-Shot Learners
  • Low-rank Prompt Interaction for Continual Vision-Language Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Learning Transferable Visual Models From Natural Language Supervision cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
  • Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
  • Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language Models cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language Models cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
  • Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language Models cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. Low-rank Prompt Interaction for Continual Vision-Language RetrievalWeicai Yan, Ye Wang, Wang Lin, Zirun Guo et al.ACM MM 2024 · 8 citations
  2. Link-Context Learning for Multimodal LLMsYan Tai, Weichen Fan, Zhao Zhang, Ziwei LiuCVPR 2024 · 7 citations
  3. Advancing Prompt Learning through an External LayerFangming Cui, Xun Yang, Chao Wu, Liang Xiao et al.ACM MM 2024 · 3 citations
  4. Adaptive Prompt-Based Semantic Embedding with Inspire Potential of Implicit Knowledge for Cross-Modal RetrievalXin Huang, Shilong Wang, Tong Jia, Zhihang Gou et al.AAAI 2025 · 2 citations
  5. Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language ModelsYongjin Yang, Jongwoo Ko, Se-Young YunEMNLP 2024 · 1 citation

Builds on20

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  3. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  4. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
  5. The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution GeneralizationDan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath et al.ICCV 2021 · 2,294 citations

Related papers

  1. APoLLo : Unified Adapter and Prompt Learning for Vision Language ModelsSanjoy Chowdhury, Sayan Nag, Dinesh ManochaEMNLP 2023 · 17 citations
  2. ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided OptimizationHao Wang, Fang Liu, Licheng Jiao, Jiahao Wang et al.AAAI 2024 · 54 citations
  3. MaPLe: Multi-modal Prompt LearningMuhammad Uzair Khattak, Hanoona Abdul Rasheed, Muhammad Maaz, Salman H. Khan et al.CVPR 2023
  4. VaMP: Variational Multi-Modal Prompt Learning for Vision-Language ModelsSilin Cheng, Kai HanNeurIPS 2025 · 7 citations
  5. CPL: Counterfactual Prompt Learning for Vision and Language ModelsXuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu et al.EMNLP 2022 · 13 citations
Dusk over the sea between two cliffs drawn in fine vertical lines