ACL Findings2025
AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularity
Zhibin Lan, Liqiang Niu, Fandong Meng, Wenbo Li, Jie Zhou, Jinsong Su
Abstract
Recently, large multimodal models (LMMs) have achieved significant advancements.When dealing with high-resolution images, dominant LMMs typically divide them into multiple local images and a global image, leading to a large number of visual tokens.In this work, we introduce AVG-LLaVA, an LMM that can adaptively select the appropriate visual granularity based on the input image and instruction.Specifically, we first apply the multiple pooling layers to obtain visual tokens at different granularities.Then we propose a visual granularity router, which includes a Transformer layer, an MLP layer, and a voter layer, used to select the appropriate visual granularity based on the image and instruction.Furthermore, we put forward RGLF, a novel training paradigm that aims at aligning the granularity predicted by the router with the preferences of the LMM, without the need for additional manually annotated data.Extensive experiments and analysis show that AVG-LLaVA achieves superior performance across 11 benchmarks, as well as significantly reduces the number of visual tokens and speeds up inference (e.g., an 85.3% reduction in visual tokens and a 2.53× increase in inference speed on the AI2D benchmark).Our code and model can be found at https: //github.com/DeepLearnXMU/AVG-LLaVA.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- Augmenting Intra-Modal Understanding in MLLMs for Robust Multimodal Keyphrase Generation cites Visual Instruction Tuning
- Augmenting Intra-Modal Understanding in MLLMs for Robust Multimodal Keyphrase Generation cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
- TOP-RL: Task-Optimized Progressive Token Pruning with Reinforcement Learning for Vision Language Models cites Visual Instruction Tuning
- VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions cites Learning Transferable Visual Models From Natural Language Supervision
- VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- Visual Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding cites Visual Instruction Tuning
- MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding cites Learning Transferable Visual Models From Natural Language Supervision
- MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings cites Visual Instruction Tuning
- UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers5
- UME-R1: Exploring Reasoning-Driven Generative Multimodal EmbeddingsZhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou et al.ICLR 2026 · 38 citations
- MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal UnderstandingXin Jin, Siyuan Li, Siyong Jian, Kai Yu et al.ICLR 2026 · 14 citations
- Augmenting Intra-Modal Understanding in MLLMs for Robust Multimodal Keyphrase GenerationJiajun Cao, Qinggang Zhang, Yunbo Tang, Zhishang Xiang et al.AAAI 2026
- TOP-RL: Task-Optimized Progressive Token Pruning with Reinforcement Learning for Vision Language ModelsHengyi Wang, Weiying Xie, Hui Jiang, Yaotao Wei et al.AAAI 2026
- VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactionsAdrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas, Yassine Ouali et al.CVPR 2026
Builds on18
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations
Related papers
- Matryoshka Multimodal ModelsMu Cai, Jianwei Yang, Jianfeng Gao, Yong Jae LeeICLR 2025
- LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMsHaoran Lou, Chunxiao Fan, Ziyan Liu, Yuexin Wu et al.ICCV 2025 · 1 citation
- Accelerating Multimodal Large Language Models by Searching Optimal Vision Token ReductionShiyu Zhao, Zhenting Wang, Felix Juefei-Xu, Xide Xia et al.CVPR 2025
- Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language ModelsGen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng et al.ICLR 2025
- VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs InferencePengfei Jiang, Hanjun Li, Linglan Zhao, Fei Chao et al.ACM MM 2025
