ACL Findings2023
Deeply Coupled Cross-Modal Prompt Learning
Xuejing Liu, Wei Tang, Jinghui Lu, Rui Zhao, Zhaojun Guo, Fei Tan
Abstract
Recent advancements in multimodal foundation models (e.g., CLIP) have excelled in zeroshot generalization. Prompt tuning involved in the knowledge transfer from foundation models to downstream tasks has gained significant attention recently. Existing prompttuning methods in cross-modal learning, however, either solely focus on language branch, or learn vision-language interaction in a shallow mechanism. In this context, we propose a Deeply coupled Cross-modal Prompt learning (DCP) method based on CLIP. DCP flexibly accommodates the interplay between vision and language with a Cross-Modal Prompt Attention (CMPA) mechanism, which enables the mutual exchange of respective representation through a well-connected multi-head attention module progressively and strongly. We then conduct comprehensive few-shot learning experiments on 11 image classification datasets and analyze the robustness to domain shift as well. Thorough experimental analysis evidently demonstrates the superb few-shot generalization and compelling domain adaption capacity of a well-executed DCP. The code can be found at https://github.com/GingL/CMPA.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Advancing Prompt Learning through an External Layer cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Advancing Prompt Learning through an External Layer cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
- Advancing Prompt Learning through an External Layer cites Learning Transferable Visual Models From Natural Language Supervision
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
- Adaptive Prompt-Based Semantic Embedding with Inspire Potential of Implicit Knowledge for Cross-Modal Retrieval cites Language Models are Few-Shot Learners
- Adaptive Prompt-Based Semantic Embedding with Inspire Potential of Implicit Knowledge for Cross-Modal Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
- Link-Context Learning for Multimodal LLMs cites Language Models are Few-Shot Learners
- Link-Context Learning for Multimodal LLMs cites Learning Transferable Visual Models From Natural Language Supervision
- Low-rank Prompt Interaction for Continual Vision-Language Retrieval cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Low-rank Prompt Interaction for Continual Vision-Language Retrieval cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Low-rank Prompt Interaction for Continual Vision-Language Retrieval cites Language Models are Few-Shot Learners
- Low-rank Prompt Interaction for Continual Vision-Language Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
- Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Learning Transferable Visual Models From Natural Language Supervision cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
- Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
- Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language Models cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
- Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language Models cites The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
- Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language Models cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers6
- Low-rank Prompt Interaction for Continual Vision-Language RetrievalWeicai Yan, Ye Wang, Wang Lin, Zirun Guo et al.ACM MM 2024 · 8 citations
- Link-Context Learning for Multimodal LLMsYan Tai, Weichen Fan, Zhao Zhang, Ziwei LiuCVPR 2024 · 7 citations
- Advancing Prompt Learning through an External LayerFangming Cui, Xun Yang, Chao Wu, Liang Xiao et al.ACM MM 2024 · 3 citations
- Adaptive Prompt-Based Semantic Embedding with Inspire Potential of Implicit Knowledge for Cross-Modal RetrievalXin Huang, Shilong Wang, Tong Jia, Zhihang Gou et al.AAAI 2025 · 2 citations
- Towards Difficulty-Agnostic Efficient Transfer Learning for Vision-Language ModelsYongjin Yang, Jongwoo Ko, Se-Young YunEMNLP 2024 · 1 citation
Builds on20
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
- The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution GeneralizationDan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath et al.ICCV 2021 · 2,294 citations
Related papers
- APoLLo : Unified Adapter and Prompt Learning for Vision Language ModelsSanjoy Chowdhury, Sayan Nag, Dinesh ManochaEMNLP 2023 · 17 citations
- ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided OptimizationHao Wang, Fang Liu, Licheng Jiao, Jiahao Wang et al.AAAI 2024 · 54 citations
- MaPLe: Multi-modal Prompt LearningMuhammad Uzair Khattak, Hanoona Abdul Rasheed, Muhammad Maaz, Salman H. Khan et al.CVPR 2023
- VaMP: Variational Multi-Modal Prompt Learning for Vision-Language ModelsSilin Cheng, Kai HanNeurIPS 2025 · 7 citations
- CPL: Counterfactual Prompt Learning for Vision and Language ModelsXuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu et al.EMNLP 2022 · 13 citations
