Lune

EMNLP Findings2021

Cross-Modal Retrieval Augmentation for Multi-Modal Classification

Shir Gur, Natalia Neverova, Christopher Stauffer, Ser-Nam Lim, Douwe Kiela, Austin Reiter

Abstract

Recent advances in using retrieval components over external knowledge sources have shown impressive results for a variety of downstream tasks in natural language processing. Here, we explore the use of unstructured external knowledge sources of images and their corresponding captions for improving visual question answering (VQA). First, we train a novel alignment model for embedding images and captions in the same space, which achieves substantial improvements in performance on image-caption retrieval w.r.t. similar methods. Second, we show that retrieval-augmented multi-modal transformers using the trained alignment model improve results on VQA over strong baselines. We further conduct extensive experiments to establish the promise of this approach, and examine novel applications for inference time such as hot-swapping indices.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Retrieval-Augmented Multimodal Language Modeling cites Learning Transferable Visual Models From Natural Language Supervision
  • Retrieval-Augmented Multimodal Language Modeling cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • Efficient Nearest Neighbor Emotion Classification with BERT-whitening cites Retrieval Augmented Language Model Pre-Training
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
  • Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gap cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality Gap cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. Retrieval-Augmented Multimodal Language ModelingMichihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James et al.ICML 2023 · 153 citations
  2. Learning Multimodal Data Augmentation in Feature SpaceZichang Liu, Zhiqiang Tang, Xingjian Shi, Aston Zhang et al.ICLR 2023 · 9 citations
  3. Efficient Nearest Neighbor Emotion Classification with BERT-whiteningWenbiao Yin, Lin ShangEMNLP 2022 · 7 citations
  4. Borrowing Human Senses: Comment-Aware Self-Training for Social Media Multimodal ClassificationChunpu Xu, Jing LiEMNLP 2022 · 3 citations
  5. Multimodal Unsupervised Domain Generalization by Retrieving Across the Modality GapChristopher Liao, Christian So, Theodoros Tsiligkaridis, Brian KulisICLR 2025

Builds on12

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  3. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
  4. Retrieval Augmented Language Model Pre-TrainingKelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat et al.ICML 2020 · 2,937 citations
  5. VL-BERT: Pre-training of Generic Visual-Linguistic RepresentationsWeijie Su, Xizhou Zhu, Yue Cao, Bin Li et al.ICLR 2020 · 1,825 citations

Related papers

  1. Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question AnsweringWeizhe Lin, Jinghong Chen, Jingbiao Mei, Alexandru Coca et al.NeurIPS 2023 · 108 citations
  2. Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search EnginesXinwei Long, Zhiyuan Ma, Ermo Hua, Kaiyan Zhang et al.AAAI 2025 · 18 citations
  3. Reveal: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge MemoryZiniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang et al.CVPR 2023
  4. Combo of Thinking and Observing for Outside-Knowledge VQAQingyi Si, Yuchen Mo, Zheng Lin, Huishan Ji et al.ACL 2023 · 7 citations
  5. Query and Attention Augmentation for Knowledge-Based Explainable ReasoningYifeng Zhang, Ming Jiang, Qi ZhaoCVPR 2022 · 14 citations
Dusk over the sea between two cliffs drawn in fine vertical lines