ACL Findings2024
Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data
Yanda Li, Chi Zhang, Gang Yu, Wanqi Yang, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, Yunchao Wei
Abstract
A: " Elephants, being wild animals, don't have the need or capability to wear graduation gowns and hold diplomas, making this image both surreal and abnormal." Prompt: "elephant wearing a graduation gown and holding a diploma …" … Q: "Can you teach me how to prepare a classic spaghetti dish? This is a picture of the ingredients I have." A: " Great! First, boil a pot of water and add spaghetti to cook." Q: "Oaky, the water is boiling and I added the spaghetti. What should I do next?" A: " Add the chopped tomatoes to the pan util the tomatoes have softened and created a sauce. Then add salt and pepper." Q: "The tomatoes have softened and I added salt and pepper. What is next?" A: " Once the spaghetti is cooked, drain it and add it to the pan with tomato sauce. And let it cook for a few more minutes." Q: "Is there any inference to be made concerning the link between two images?" A: "Watering the flowers with the garden's watering can makes the flowers bloom beautifully." Q: "Okay, I've cooked it, here's the spaghetti." A: " Great! You did a good job." Q: "What profession does the person in the image represent?"
A: "Carpenter. As he is in a woodshop surrounded by woodworking tools and wood shavings.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- NExT-GPT: Any-to-Any Multimodal LLM cites High-Resolution Image Synthesis with Latent Diffusion Models
- NExT-GPT: Any-to-Any Multimodal LLM cites Visual Instruction Tuning
- NExT-GPT: Any-to-Any Multimodal LLM cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- NExT-GPT: Any-to-Any Multimodal LLM cites Learning Transferable Visual Models From Natural Language Supervision
- NExT-GPT: Any-to-Any Multimodal LLM cites Flamingo: a Visual Language Model for Few-Shot Learning
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models cites Visual Instruction Tuning
- Charts Are Not Images: On the Challenges of Scientific Chart Editing cites High-Resolution Image Synthesis with Latent Diffusion Models
- Charts Are Not Images: On the Challenges of Scientific Chart Editing cites Learning Transferable Visual Models From Natural Language Supervision
- Visual Instruction Tuning cites High-Resolution Image Synthesis with Latent Diffusion Models
- Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs cites High-Resolution Image Synthesis with Latent Diffusion Models
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs cites Visual Instruction Tuning
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs cites Learning Transferable Visual Models From Natural Language Supervision
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
- Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers4
- NExT-GPT: Any-to-Any Multimodal LLMShengqiong Wu, Hao Fei, Leigang Qu, Wei Ji et al.ICML 2024 · 786 citations
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMsLing Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu et al.ICML 2024 · 231 citations
- Charts Are Not Images: On the Challenges of Scientific Chart EditingShawn Li, Ryan Rossi, Sungchul Kim, Sunav Choudhary et al.ICLR 2026 · 11 citations
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language ModelsShojiro Yamabe, Futa Waseda, Daiki Shiono, Tsubasa TakahashiCVPR 2026 · 2 citations
Builds on15
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
Related papers
- SEED-Bench: Benchmarking Multimodal Large Language ModelsBohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang et al.CVPR 2024
- UniGen-1.5: Enhancing Image Generation and Editing through Reward Unification in RLRui Tian, Mingfei Gao, Haiming Gang, Jiasen Lu et al.CVPR 2026
- End-to-End Learning of Visual Representations From Uncurated Instructional VideosAntoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev et al.CVPR 2020
- Images Speak in Images: A Generalist Painter for In-Context Visual LearningXinlong Wang, Wen Wang, Yue Cao, Chunhua Shen et al.CVPR 2023
- Text-Driven Image Editing via Learnable RegionsYuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Lu Jiang et al.CVPR 2024
