Data-efficient Large Vision Models through Sequential Autoregression
Zhiwei Hao, Jianyuan Guo, Chengcheng Wang, Yehui Tang, Han Wu, Han Hu, Kai Han, Chang Xu
Abstract
Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding. These models are intended to not only comprehend but also seamlessly transit to out-of-domain tasks. However, current endeavors are hamstrung by an over-reliance on colossal models, exemplified by models with upwards of 3B parameters, and the necessity for an extensive corpus of visual data, often comprising a staggering 400B tokens (Bai et al., 2023) . In this paper, we delve into the development of an efficient, autoregression-based vision model, innovatively architected to operate on a limited dataset. We meticulously demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding during the testing phase. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint, and a marked decrease in training data requirements, thereby paving the way for more sustainable and accessible advancements in the field of generalist vision models. The code is available at https://github.com/ggjy/DeLVM .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
8 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Is Visual in-Context Learning for Compositional Medical Tasks Within Reach? cites Language Models are Few-Shot Learners
- Segment Anything cites Language Models are Few-Shot Learners
- QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor Expansion cites Enhancing Vision Transformer: Amplifying Non-Linearity in Feedforward Network Module
Cited by top-tier papers8
- EgoAgent: A Joint Predictive Agent Model in Egocentric WorldsLu Chen, Yizhou Wang, Shixiang Tang, Qianhong Ma et al.ICCV 2025 · 10 citations
- QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor ExpansionYixing Xu, Chao Li, Dong Li, Xiao Sheng et al.NeurIPS 2024 · 7 citations
- Enhancing Vision Transformer: Amplifying Non-Linearity in Feedforward Network ModuleYixing Xu, Chao Li, Dong Li, Xiao Sheng et al.ICML 2024 · 5 citations
- Hybrid Data-Free Knowledge DistillationJialiang Tang, Shuo Chen, Chen GongAAAI 2025 · 2 citations
- Is Visual in-Context Learning for Compositional Medical Tasks Within Reach?Simon Reiß, Zdravko Marinov, Alexander Jaus, Constantin Seibold et al.ICCV 2025
Builds on20
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao et al.ICCV 2023 · 13,211 citations
- Generative Pretraining From PixelsMark Chen, Alec Radford, Rewon Child, Jeffrey Wu et al.ICML 2020 · 1,773 citations
- Decoupling Representation and Classifier for Long-Tailed RecognitionBingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan et al.ICLR 2020 · 1,496 citations
- Contrastive Representation DistillationYonglong Tian, Dilip Krishnan, Phillip IsolaICLR 2020 · 1,305 citations
Related papers
- Sequential Modeling Enables Scalable Learning for Large Vision ModelsYutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar et al.CVPR 2024
- X-Prompt: Generalizable Auto-Regressive Visual Learning with In-Context PromptingZeyi Sun, Ziyang Chu, Pan Zhang, Tong Wu et al.ICCV 2025 · 1 citation
- Unveiling Encoder-Free Vision-Language ModelsHaiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang et al.NeurIPS 2024 · 107 citations
- Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot GeneralizationYang Shen, Xiu-Shen Wei, Yifan Sun, Yuxin Song et al.ICML 2025
- Eve: Efficient Multimodal Vision Language Models with Elastic Visual ExpertsMiao Rang, Zhenni Bi, Chuanjian Liu, Yehui Tang et al.AAAI 2025 · 16 citations
