Lune

ICML2023Top-tier venue

Retrieval-Augmented Multimodal Language Modeling

Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, Wen-Tau Yih

Abstract

Recent multimodal models such as DALL-E and CM3 have achieved remarkable progress in textto-image and image-to-text generation. However, these models store all learned knowledge (e.g., the appearance of the Eiffel Tower) in the model parameters, requiring increasingly larger models and training data to capture more knowledge. To integrate knowledge in a more scalable and modular way, we propose a retrievalaugmented multimodal model, which enables a base multimodal model (generator) to refer to relevant knowledge fetched by a retriever from external memory (e.g., multimodal documents on the web). Specifically, we implement a retriever using the pretrained CLIP model and a generator using the CM3 Transformer architecture, and train this model using the LAION dataset. Our resulting model, named Retrieval-Augmented CM3 (RA-CM3), is the first multimodal model that can retrieve and generate mixtures of text and images. We show that RA-CM3 significantly outperforms baseline multimodal models such as DALL-E and CM3 on both image and caption generation tasks (12 FID and 17 CIDEr improvements on MSCOCO), while requiring much less compute for training (<30% of DALL-E). Moreover, we show that RA-CM3 exhibits novel capabilities such as knowledge-intensive image generation and multimodal in-context learning

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

60 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Language Is Not All You Need: Aligning Perception with Language Models cites Learning Transferable Visual Models From Natural Language Supervision
  • Large Language Models as Analogical Reasoners cites Language Models are Few-Shot Learners
  • Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models cites Language Models are Few-Shot Learners
  • Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models cites Large Language Models as Analogical Reasoners
  • Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models cites Language Models are Few-Shot Learners
  • Language Models are Few-Shot Learners cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • Lift Yourself Up: Retrieval-augmented Text Generation with Self-Memory cites Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
  • Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
  • Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites Language Models are Few-Shot Learners
  • Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers60

  1. Language Is Not All You Need: Aligning Perception with Language ModelsShaohan Huang, Li Dong, Wenhui Wang, Yaru Hao et al.NeurIPS 2023 · 810 citations
  2. Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language ModelsYushi Hu, Weijia Shi, Xingyu Fu, Dan Roth et al.NeurIPS 2024 · 373 citations
  3. Lift Yourself Up: Retrieval-augmented Text Generation with Self-MemoryXin Cheng, Di Luo, Xiuying Chen, Lemao Liu et al.NeurIPS 2023 · 177 citations
  4. Large Language Models as Analogical ReasonersMichihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat et al.ICLR 2024 · 155 citations
  5. Buffer of Thoughts: Thought-Augmented Reasoning with Large Language ModelsLing Yang, Zhaochen Yu, Tianjun Zhang, Shiyi Cao et al.NeurIPS 2024 · 144 citations

Builds on22

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  3. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  4. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  5. Photorealistic Text-to-Image Diffusion Models with Deep Language UnderstandingChitwan Saharia, William Chan, Saurabh Saxena, Lala Li et al.NeurIPS 2022 · 8,965 citations

Related papers

  1. MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and TextWenhu Chen, Hexiang Hu, Xi Chen, Pat Verga et al.EMNLP 2022 · 89 citations
  2. Reveal: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge MemoryZiniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang et al.CVPR 2023
  3. Towards Language-Free Training for Text-to-Image GenerationYufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li et al.CVPR 2022 · 182 citations
  4. Boosting Knowledge Utilization in Multimodal Large Language Models via Adaptive Logits Fusion and Attention ReallocationWenbin An, Jiahao Nie, Feng Tian, Haonan Lin et al.NeurIPS 2025 · 4 citations
  5. Retrieval-Enhanced Contrastive Vision-Text ModelsAhmet Iscen, Mathilde Caron, Alireza Fathi, Cordelia SchmidICLR 2024 · 44 citations
Dusk over the sea between two cliffs drawn in fine vertical lines