Lune

EMNLP Findings2022

Conditioned Masked Language and Image Modeling for Image-Text Dense Retrieval

Ziyang Luo, Yadong Xi, Rongsheng Zhang, GongZheng Li, Zeng Zhao, Jing Ma

Abstract

Image-text retrieval is a fundamental crossmodal task that takes image/text as a query to retrieve relevant data of another type. The large-scale two-stream pre-trained models like CLIP have achieved tremendous success in this area. They embed the images and texts into instance representations with two separate encoders, aligning them on the instance-level with contrastive learning. Beyond this, the following works adopt the fine-grained token-level interaction (Masked Language and Image Modeling) to boost performance further. However, the vanilla token-level objectives are not designed to aggregate the image-text alignment information into the instance representations, but the token representations, causing a gap between pre-training and application. To address this issue, we carefully design two novel conditioned token-level pre-training objectives, Conditioned Masked Language and Image Modeling (ConMLM and ConMIM), forcing models to aggregate the token-level alignment information into the instance representations. Combing with the instance-level contrastive learning, we propose our cross-modal dense retrieval framework, Conditioned Language-Image Pretraining (ConLIP). Experimental results on two popular cross-modal retrieval benchmarks (MSCOCO and Flickr30k) reveal the effectiveness of our methods.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Deberta: decoding-Enhanced Bert with Disentangled Attention cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
  • OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling cites Learning Transferable Visual Models From Natural Language Supervision
  • Seeing What's Not There: Negation Understanding Needs More Than Training cites Learning Transferable Visual Models From Natural Language Supervision
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
  • LexLIP: Lexicon-Bottlenecked Language-Image Pre-Training for Large-Scale Image-Text Sparse Retrieval cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • LexLIP: Lexicon-Bottlenecked Language-Image Pre-Training for Large-Scale Image-Text Sparse Retrieval cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • LexLIP: Lexicon-Bottlenecked Language-Image Pre-Training for Large-Scale Image-Text Sparse Retrieval cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked AutoencodersAnthony Fuller, Koreen Millard, James R. GreenNeurIPS 2023 · 245 citations
  2. OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring ModelingLinhui Xiao, Xiaoshan Yang, Fang Peng, Yaowei Wang et al.NeurIPS 2024 · 45 citations
  3. LexLIP: Lexicon-Bottlenecked Language-Image Pre-Training for Large-Scale Image-Text Sparse RetrievalZiyang Luo, Pu Zhao, Can Xu, Xiubo Geng et al.ICCV 2023 · 30 citations
  4. Seeing What's Not There: Negation Understanding Needs More Than TrainingBhuvan Aggarwal, Amit More, Mudit Soni, Srinivasa Divakar BhatICLR 2026

Builds on18

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  4. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
  5. Deberta: decoding-Enhanced Bert with Disentangled AttentionPengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu ChenICLR 2021 · 3,729 citations

Related papers

  1. COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal RetrievalHaoyu Lu, Nanyi Fei, Yuqi Huo, Yizhao Gao et al.CVPR 2022 · 68 citations
  2. Overcoming the Pitfalls of Vision-Language Model for Image-Text RetrievalFeifei Zhang, Sijia Qu, Fan Shi, Changsheng XuACM MM 2024 · 12 citations
  3. Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality InversionMarco Mistretta, Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini et al.ICLR 2025
  4. Target-Guided Composed Image RetrievalHaokun Wen, Xian Zhang, Xuemeng Song, Yinwei Wei et al.ACM MM 2023 · 53 citations
  5. DenseCLIP: Language-Guided Dense Prediction with Context-Aware PromptingYongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang et al.CVPR 2022 · 527 citations
Dusk over the sea between two cliffs drawn in fine vertical lines