Lune

ACL Findings2024

Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data

Yanda Li, Chi Zhang, Gang Yu, Wanqi Yang, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, Yunchao Wei

Abstract

A: " Elephants, being wild animals, don't have the need or capability to wear graduation gowns and hold diplomas, making this image both surreal and abnormal." Prompt: "elephant wearing a graduation gown and holding a diploma …" … Q: "Can you teach me how to prepare a classic spaghetti dish? This is a picture of the ingredients I have." A: " Great! First, boil a pot of water and add spaghetti to cook." Q: "Oaky, the water is boiling and I added the spaghetti. What should I do next?" A: " Add the chopped tomatoes to the pan util the tomatoes have softened and created a sauce. Then add salt and pepper." Q: "The tomatoes have softened and I added salt and pepper. What is next?" A: " Once the spaghetti is cooked, drain it and add it to the pan with tomato sauce. And let it cook for a few more minutes." Q: "Is there any inference to be made concerning the link between two images?" A: "Watering the flowers with the garden's watering can makes the flowers bloom beautifully." Q: "Okay, I've cooked it, here's the spaghetti." A: " Great! You did a good job." Q: "What profession does the person in the image represent?"

A: "Carpenter. As he is in a woodshop surrounded by woodworking tools and wood shavings.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • NExT-GPT: Any-to-Any Multimodal LLM cites High-Resolution Image Synthesis with Latent Diffusion Models
  • NExT-GPT: Any-to-Any Multimodal LLM cites Visual Instruction Tuning
  • NExT-GPT: Any-to-Any Multimodal LLM cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • NExT-GPT: Any-to-Any Multimodal LLM cites Learning Transferable Visual Models From Natural Language Supervision
  • NExT-GPT: Any-to-Any Multimodal LLM cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models cites Visual Instruction Tuning
  • Charts Are Not Images: On the Challenges of Scientific Chart Editing cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Charts Are Not Images: On the Challenges of Scientific Chart Editing cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs cites High-Resolution Image Synthesis with Latent Diffusion Models
  • Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs cites Visual Instruction Tuning
  • Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs cites Learning Transferable Visual Models From Natural Language Supervision
  • BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. NExT-GPT: Any-to-Any Multimodal LLMShengqiong Wu, Hao Fei, Leigang Qu, Wei Ji et al.ICML 2024 · 786 citations
  2. Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMsLing Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu et al.ICML 2024 · 231 citations
  3. Charts Are Not Images: On the Challenges of Scientific Chart EditingShawn Li, Ryan Rossi, Sungchul Kim, Sunav Choudhary et al.ICLR 2026 · 11 citations
  4. Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language ModelsShojiro Yamabe, Futa Waseda, Daiki Shiono, Tsubasa TakahashiCVPR 2026 · 2 citations

Builds on15

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
  3. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  4. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  5. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations

Related papers

  1. SEED-Bench: Benchmarking Multimodal Large Language ModelsBohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang et al.CVPR 2024
  2. UniGen-1.5: Enhancing Image Generation and Editing through Reward Unification in RLRui Tian, Mingfei Gao, Haiming Gang, Jiasen Lu et al.CVPR 2026
  3. End-to-End Learning of Visual Representations From Uncurated Instructional VideosAntoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev et al.CVPR 2020
  4. Images Speak in Images: A Generalist Painter for In-Context Visual LearningXinlong Wang, Wen Wang, Yue Cao, Chunhua Shen et al.CVPR 2023
  5. Text-Driven Image Editing via Learnable RegionsYuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Lu Jiang et al.CVPR 2024
Dusk over the sea between two cliffs drawn in fine vertical lines