Lune

ICML2024Top-tier venue

Data-efficient Large Vision Models through Sequential Autoregression

Zhiwei Hao, Jianyuan Guo, Chengcheng Wang, Yehui Tang, Han Wu, Han Hu, Kai Han, Chang Xu

Abstract

Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding. These models are intended to not only comprehend but also seamlessly transit to out-of-domain tasks. However, current endeavors are hamstrung by an over-reliance on colossal models, exemplified by models with upwards of 3B parameters, and the necessity for an extensive corpus of visual data, often comprising a staggering 400B tokens (Bai et al., 2023) . In this paper, we delve into the development of an efficient, autoregression-based vision model, innovatively architected to operate on a limited dataset. We meticulously demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding during the testing phase. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint, and a marked decrease in training data requirements, thereby paving the way for more sustainable and accessible advancements in the field of generalist vision models. The code is available at https://github.com/ggjy/DeLVM .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

8 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Is Visual in-Context Learning for Compositional Medical Tasks Within Reach? cites Language Models are Few-Shot Learners
  • Segment Anything cites Language Models are Few-Shot Learners
  • QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor Expansion cites Enhancing Vision Transformer: Amplifying Non-Linearity in Feedforward Network Module
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers8

  1. EgoAgent: A Joint Predictive Agent Model in Egocentric WorldsLu Chen, Yizhou Wang, Shixiang Tang, Qianhong Ma et al.ICCV 2025 · 10 citations
  2. QT-ViT: Improving Linear Attention in ViT with Quadratic Taylor ExpansionYixing Xu, Chao Li, Dong Li, Xiao Sheng et al.NeurIPS 2024 · 7 citations
  3. Enhancing Vision Transformer: Amplifying Non-Linearity in Feedforward Network ModuleYixing Xu, Chao Li, Dong Li, Xiao Sheng et al.ICML 2024 · 5 citations
  4. Hybrid Data-Free Knowledge DistillationJialiang Tang, Shuo Chen, Chen GongAAAI 2025 · 2 citations
  5. Is Visual in-Context Learning for Compositional Medical Tasks Within Reach?Simon Reiß, Zdravko Marinov, Alexander Jaus, Constantin Seibold et al.ICCV 2025

Builds on20

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao et al.ICCV 2023 · 13,211 citations
  3. Generative Pretraining From PixelsMark Chen, Alec Radford, Rewon Child, Jeffrey Wu et al.ICML 2020 · 1,773 citations
  4. Decoupling Representation and Classifier for Long-Tailed RecognitionBingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan et al.ICLR 2020 · 1,496 citations
  5. Contrastive Representation DistillationYonglong Tian, Dilip Krishnan, Phillip IsolaICLR 2020 · 1,305 citations

Related papers

  1. Sequential Modeling Enables Scalable Learning for Large Vision ModelsYutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar et al.CVPR 2024
  2. X-Prompt: Generalizable Auto-Regressive Visual Learning with In-Context PromptingZeyi Sun, Ziyang Chu, Pan Zhang, Tong Wu et al.ICCV 2025 · 1 citation
  3. Unveiling Encoder-Free Vision-Language ModelsHaiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang et al.NeurIPS 2024 · 107 citations
  4. Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot GeneralizationYang Shen, Xiu-Shen Wei, Yifan Sun, Yuxin Song et al.ICML 2025
  5. Eve: Efficient Multimodal Vision Language Models with Elastic Visual ExpertsMiao Rang, Zhenni Bi, Chuanjian Liu, Yehui Tang et al.AAAI 2025 · 16 citations
Dusk over the sea between two cliffs drawn in fine vertical lines