Lune

ACL Findings2024

Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection

Ruibo Chen, Yihan Wu, Lichang Chen, Guodong Liu, Qi He, Tianyi Xiong, Chenxi Liu, Junfeng Guo, Heng Huang

Abstract

Data selection in instruction tuning emerges as a pivotal process for acquiring high-quality data and training instruction-following large language models (LLMs), but it is still a new and unexplored research area for vision-language models (VLMs). Existing data selection approaches on LLMs either rely on single unreliable scores, or use downstream tasks for selection, which is time-consuming and can lead to potential over-fitting on the chosen evaluation datasets. To address this challenge, we introduce a novel dataset selection method, SELF-FILTER, that utilizes the VLM itself as a filter. This approach is inspired by the observation that VLMs benefit from training with the most challenging instructions. SELF-FILTER operates in two stages. In the first stage, we devise a scoring network to evaluate the difficulty of training instructions, which is cotrained with the VLM. In the second stage, we use the trained score net to measure the difficulty of each instruction, select the most challenging samples, and penalize similar samples to encourage diversity. Comprehensive experiments on LLaVA and MiniGPT-4 show that SELF-FILTER can reach better results compared to full data settings with merely about 15% samples, and can achieve superior performance against competitive baselines. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

17 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement cites Visual Instruction Tuning
  • SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement cites Learning Transferable Visual Models From Natural Language Supervision
  • CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization cites Visual Instruction Tuning
  • CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity Optimization cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites Training language models to follow instructions with human feedback
  • BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models cites Visual Instruction Tuning
  • SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language Models cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers17

  1. SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-ImprovementXiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu et al.NeurIPS 2025 · 158 citations
  2. A Resilient and Accessible Distribution-Preserving Watermark for Large Language ModelsYihan Wu, Zhengmian Hu, Junfeng Guo, Hongyang Zhang et al.ICML 2024 · 50 citations
  3. ZeroMark: Towards Dataset Ownership Verification without Disclosing WatermarkJunfeng Guo, Yiming Li, Ruibo Chen, Yihan Wu et al.NeurIPS 2024 · 24 citations
  4. CoIDO: Efficient Data Selection for Visual Instruction Tuning via Coupled Importance-Diversity OptimizationYichen Yan, Ming Zhong, Qi Zhu, Xiaoling Gu et al.NeurIPS 2025 · 8 citations
  5. SpatiaLQA: A Benchmark for Evaluating Spatial Logical Reasoning in Vision-Language ModelsYuechen Xie, Xiaoyan Zhang, Yicheng Shan, Zhu Hao et al.CVPR 2026 · 8 citations

Builds on24

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  4. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  5. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations

Related papers

  1. Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-TuningMing Li, Yong Zhang, Shwai He, Zhitao Li et al.ACL 2024 · 16 citations
  2. T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction TuningYanjun Fu, Faisal Hamman, Sanghamitra DuttaNeurIPS 2025 · 15 citations
  3. Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction TuningBardia Safaei, Faizan Siddiqui, Jiacong Xu, Vishal M. Patel et al.CVPR 2025
  4. CPQS-Tuning: A Model Self-Perception-Based Data Filtering Algorithm for Efficient Instruction Fine-TuningYI Ren, Yanhui Li, Tianyi Zhang, Diandong LiuICLR 2026
  5. Mastering Collaborative Multi-Modal Data Selection: A Focus on Informativeness, Uniqueness, and RepresentativenessQifan Yu, Zhebei Shen, Zhongqi Yue, Yang Wu et al.ICCV 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines