Lune

EMNLP2024Top-tier venue

World to Code: Multi-modal Data Generation via Self-Instructed Compositional Captioning and Filtering

Jiacong Wang, Bohong Wu, Haiyong Jiang, Xun Zhou, Xin Xiao, Haoyuan Guo, Jun Xiao

Abstract

Recent advances in Vision-Language Models (VLMs) and the scarcity of high-quality multimodal alignment data have inspired numerous researches on synthetic VLM data generation. The conventional norm in VLM data construction uses a mixture of specialists in caption and OCR, or stronger VLM APIs and expensive human annotation. In this paper, we present World to Code (W2C), a meticulously curated multi-modal data construction pipeline that organizes the final generation output into a Python code format. The pipeline leverages the VLM itself to extract cross-modal information via different prompts and filter the generated outputs again via a consistency filtering strategy. Experiments have demonstrated the high quality of W2C by improving various existing visual question answering and visual grounding benchmarks across different VLMs. Further analysis also demonstrates that the new code parsing ability of VLMs presents better crossmodal equivalence than the commonly used detail caption ability. Our code is available at https://github.com/foundation-multimodalmodels/World2Code .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

8 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Scheduling Your LLM Reinforcement Learning with Reasoning Trees cites The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer
  • Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Method cites The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer
  • Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Method cites Learning Transferable Visual Models From Natural Language Supervision
  • The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer cites Learning Transferable Visual Models From Natural Language Supervision
  • The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • VGR: Visual Grounded Reasoning cites Scheduling Your LLM Reinforcement Learning with Reasoning Trees
  • VGR: Visual Grounded Reasoning cites The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer
  • VGR: Visual Grounded Reasoning cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers8

  1. VGR: Visual Grounded ReasoningJiacong Wang, Zijian Kang, Haochen Wang, Xiao Liang et al.ICLR 2026 · 64 citations
  2. Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and MethodHaochen Wang, Xiangtai Li, Zilong Huang, Anran Wang et al.ICLR 2026 · 45 citations
  3. Scheduling Your LLM Reinforcement Learning with Reasoning TreesHong Wang, Zhezheng Hao, Jian Luo, Chenxing Wei et al.ICLR 2026 · 16 citations
  4. VideoWorld 2: Learning Transferable Knowledge from Real-world VideosZhongwei Ren, Yunchao Wei, Xiao Yu, Guixun Luo et al.CVPR 2026 · 9 citations
  5. The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single TransformerWeixian Lei, Jiacong Wang, Haochen Wang, Xiangtai Li et al.ICCV 2025 · 3 citations

Builds on16

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
  4. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations
  5. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations

Related papers

  1. UnifiedVisual: A Framework for Constructing Unified Vision-Language DatasetsPengyu Wang, Shaojun Zhou, Chenkun Tan, Xinghao Wang et al.EMNLP 2025
  2. MM2SQL: A Benchmark and Method for Visually-Grounded SQL GenerationShengze Shi, Tao Ren, Li Qi, Tingrui Yang et al.ICDE 2026
  3. VIGC: Visual Instruction Generation and CorrectionBin Wang, Fan Wu, Xiao Han, Jiahui Peng et al.AAAI 2024 · 95 citations
  4. UNIMO-2: End-to-End Unified Vision-Language Grounded LearningWei Li, Can Gao, Guocheng Niu, Xinyan Xiao et al.ACL Findings 2022 · 15 citations
  5. Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data GenerationYue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta et al.ACL 2025
Dusk over the sea between two cliffs drawn in fine vertical lines