EMNLP Findings2025
ViFT: Towards Visual Instruction-Free Fine-tuning for Large Vision-Language Models
Zikang Liu, Kun Zhou, Wayne Xin Zhao, Dawei Gao, Yaliang Li, Ji-Rong Wen
Abstract
Visual instruction tuning has become the predominant technology in eliciting the multimodal task-solving capabilities of large visionlanguage models (LVLMs). Despite the success, as visual instructions require images as the input, it would leave the gap in inheriting the task-solving capabilities from the backbone LLMs, and make it costly to collect a large-scale high-quality dataset. To address it, we propose ViFT, a visual instruction-free fine-tuning framework for LVLMs. In ViFT, we only require the text-only instructions and image caption data during training, to separately learn the task-solving and visual perception abilities. During inference, we extract and combine the representations of the text and image inputs, for fusing the two abilities to fulfill multimodal tasks. Experimental results demonstrate that ViFT can achieve state-of-the-art performance on several downstream benchmarks, with rather less training data. Our code and data will be publicly released.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
- Visual Instruction Tuning cites Language Models are Few-Shot Learners
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
- Finetuned Language Models are Zero-Shot Learners cites Language Models are Few-Shot Learners
Builds on23
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
Related papers
- Learning to Instruct for Visual Instruction TuningZhihan Zhou, Feng Hong, Jiaan Luo, Yushi Ye et al.NeurIPS 2025 · 8 citations
- VILA: On Pre-training for Visual Language ModelsJi Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov et al.CVPR 2024
- Visual Compositional TuningXindi Wu, Hee Seung Hwang, Polina Kirichenko, Esin Tureci et al.ICLR 2026 · 3 citations
- Less is More: High-value Data Selection for Visual Instruction TuningZikang Liu, Kun Zhou, Wayne Xin Zhao, Dawei Gao et al.ACM MM 2025 · 3 citations
- VFA: Empowering Multilingual MLLMs via Vision-Free AdaptationYixia Li, Yaqing Shi, Zhiwen Ruan, Dongdong Zhang et al.ACL 2026
