Lune

ICLR2025Top-tier venue

ZooProbe: A Data Engine for Evaluating, Exploring, and Evolving Large-scale Training Data for Multimodal LLMs

Yi-Kai Zhang, Shiyin Lu, Qing-Guo Chen, De-Chuan Zhan, Han-Jia Ye

Abstract

Multimodal Large Language Models (MLLMs) are thriving through continuous fine-tuning by LLMs. Driven by the law that "scale is everything", MLLMs expand their training sets during version iterations. In this paper, we propose a large-scale training data engine built around an evaluating-exploring-evolving (E3) loop. Evaluating the data provides insights into its characteristics. Exploring quality rules helps identify which data enhances training. Together, these processes facilitate the systematic evolution of new, high-quality data. With the E3 loop, we introduce ZOOPROBE, an efficient data engine for MLLMs. First, the problem of data expansion is formalized as a tree of sampling and growth. ZOOPROBE introduces a small-scale model zoo to obtain comprehensive evaluations for child datasets. From multiple perspectives, visual, textual, and multimodal models cover over 50 dimensions of intrinsic and meta attributes, such as object and topic distribution, and higher-level properties, like annotation quality and scene complexity. ZOOPROBE constructs based on A ⋆ search, modeling the heuristic function as a quality estimate from data evaluation results. It dynamically explores the rule of data quality based on the model state of the probe datasets. Additionally, it evolves new targeted data with identified high-quality rules. We also develop an extra heuristic quality ranker with the data utilized and discarded during the expansion. Our experiments show that ZOOPROBE significantly breaks the scaling law in multimodal instruction fine-tuning at scales of 260k and below. ZOOPROBE generates high-quality data that accelerates MLLM training and enhances performance, automating the evolution of large-scale training data.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • Segment Anything cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Segment Anything cites Language Models are Few-Shot Learners
  • Segment Anything cites Learning Transferable Visual Models From Natural Language Supervision
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on31

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  3. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  4. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  5. Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao et al.ICCV 2023 · 13,211 citations

Related papers

  1. From Blind Spots to Gains: Diagnostic-Driven Iterative Training for Large Multimodal ModelsHongrui Jia, Chaoya Jiang, Yongrui Heng, Shikun Zhang et al.ICML 2026
  2. Advancing Multimodal Large Language Models in Chart Question Answering with Visualization-Referenced Instruction TuningXingchen Zeng, Haichuan Lin, Yilin Ye, Wei ZengIEEE VIS 2024 · 23 citations
  3. MM-IFEngine: Towards Multimodal Instruction FollowingShengyuan Ding, Shenxi Wu, Xiangyu Zhao, Yuhang Zang et al.ICCV 2025 · 3 citations
  4. LoopTool: Closing the Data-Training Loop for Robust LLM Tool CallsKangning Zhang, Weiwen Liu, Wenxiang Jiao, Kounianhua Du et al.ACL 2026 · 18 citations
  5. EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive HierarchyJinzhao Li, Yinuo Chen, Dongxu Piao, Panwang Pan et al.CVPR 2026 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines