Lune

ACL Findings2025

AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularity

Zhibin Lan, Liqiang Niu, Fandong Meng, Wenbo Li, Jie Zhou, Jinsong Su

摘要

Recently, large multimodal models (LMMs) have achieved significant advancements.When dealing with high-resolution images, dominant LMMs typically divide them into multiple local images and a global image, leading to a large number of visual tokens.In this work, we introduce AVG-LLaVA, an LMM that can adaptively select the appropriate visual granularity based on the input image and instruction.Specifically, we first apply the multiple pooling layers to obtain visual tokens at different granularities.Then we propose a visual granularity router, which includes a Transformer layer, an MLP layer, and a voter layer, used to select the appropriate visual granularity based on the image and instruction.Furthermore, we put forward RGLF, a novel training paradigm that aims at aligning the granularity predicted by the router with the preferences of the LMM, without the need for additional manually annotated data.Extensive experiments and analysis show that AVG-LLaVA achieves superior performance across 11 benchmarks, as well as significantly reduces the number of visual tokens and speeds up inference (e.g., an 85.3% reduction in visual tokens and a 2.53× increase in inference speed on the AI2D benchmark).Our code and model can be found at https: //github.com/DeepLearnXMU/AVG-LLaVA.

问 Lune

几分钟读透这篇 Paper,不用再花一下午。

Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。

已有 5 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。

可以从这些问题问起

Lune 可以查阅

全文 · 引用它的顶会 Paper · 相关的 Paper

获取答案

免费开始,无需绑卡

脉络

  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning 引用了 MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning 引用了 Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • Augmenting Intra-Modal Understanding in MLLMs for Robust Multimodal Keyphrase Generation 引用了 Visual Instruction Tuning
  • Augmenting Intra-Modal Understanding in MLLMs for Robust Multimodal Keyphrase Generation 引用了 MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • TOP-RL: Task-Optimized Progressive Token Pruning with Reinforcement Learning for Vision Language Models 引用了 Visual Instruction Tuning
  • VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions 引用了 Learning Transferable Visual Models From Natural Language Supervision
  • VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions 引用了 Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • Visual Instruction Tuning 引用了 Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning 引用了 Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding 引用了 Visual Instruction Tuning
  • MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding 引用了 Learning Transferable Visual Models From Natural Language Supervision
  • MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understanding 引用了 Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
  • UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings 引用了 Visual Instruction Tuning
  • UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings 引用了 Learning Transferable Visual Models From Natural Language Supervision
它引用的引用它的从左到右按年份排列,引用越多,位置越高。

引用它的顶会 Paper5

  1. UME-R1: Exploring Reasoning-Driven Generative Multimodal EmbeddingsZhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou 等ICLR 2026 · 38 次引用
  2. MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal UnderstandingXin Jin, Siyuan Li, Siyong Jian, Kai Yu 等ICLR 2026 · 14 次引用
  3. Augmenting Intra-Modal Understanding in MLLMs for Robust Multimodal Keyphrase GenerationJiajun Cao, Qinggang Zhang, Yunbo Tang, Zhishang Xiang 等AAAI 2026
  4. TOP-RL: Task-Optimized Progressive Token Pruning with Reinforcement Learning for Vision Language ModelsHengyi Wang, Weiying Xie, Hui Jiang, Yaotao Wei 等AAAI 2026
  5. VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactionsAdrian Bulat, Alberto Baldrati, Ioannis Maniadis Metaxas, Yassine Ouali 等CVPR 2026

它引用的 Paper18

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh 等ICML 2021 · 47,906 次引用
  2. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 次引用
  3. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong 等NeurIPS 2023 · 4,013 次引用
  4. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li 等ICLR 2024 · 3,079 次引用
  5. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu 等NeurIPS 2022 · 2,727 次引用

相关 Paper

  1. Matryoshka Multimodal ModelsMu Cai, Jianwei Yang, Jianfeng Gao, Yong Jae LeeICLR 2025
  2. LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMsHaoran Lou, Chunxiao Fan, Ziyan Liu, Yuexin Wu 等ICCV 2025 · 1 次引用
  3. Accelerating Multimodal Large Language Models by Searching Optimal Vision Token ReductionShiyu Zhao, Zhenting Wang, Felix Juefei-Xu, Xide Xia 等CVPR 2025
  4. Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language ModelsGen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng 等ICLR 2025
  5. VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs InferencePengfei Jiang, Hanjun Li, Linglan Zhao, Fei Chao 等ACM MM 2025
黄昏的海面,两侧是细线勾勒的悬崖