CVPR Findings2026
GM-Skip: Metric-Guided Transformer Block Skipping for Efficient Vision-Language Models
Lianming Huang, Haibo Hu, Qiao Li, Xin He, Nan Guan, Chun Jason Xue
Abstract
Transformer-based Vision-Language Models (VLMs) have achieved remarkable success in image captioning, object recognition, and visual reasoning, but their heavy computational overhead limits deployment in latency-sensitive settings such as autonomous driving. We introduce GM-Skip, a flexible and metric-adaptive framework for Transformer block skipping that accelerates VLM inference while preserving task performance. GM-Skip employs a greedy, metric-guided block selection strategy that leverages feedback from evaluation metrics (e.g., accuracy, CIDEr) to identify redundant layers, combined with a reverse-order deletion mechanism that preserves essential early representations. A score-sparsity balance objective further enables controllable trade-offs between efficiency and accuracy across tasks. Experiments on COCO, CODA, and realworld driving scenarios show that GM-Skip consistently improves inference speed with minimal performance loss. On COCO, it boosts single-object classification accuracy on the Person category by 19.1%-87.3% while skipping over 40% of Transformer blocks, enhances multi-object detection by 23.4% with 35% sparsity, and doubles the captioning quality (CIDEr) of Qwen-VL under comparable latency. When integrated into an autonomous vehicle running Autoware.Universe, GM-Skip achieves up to 45.4% latency reduction in real-world inference, validating its practical effectiveness for efficient, adaptive VLM deployment.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
- Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
- Flamingo: a Visual Language Model for Few-Shot Learning cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Builds on13
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
- SparseGPT: Massive Language Models Can be Accurately Pruned in One-ShotElias Frantar, Dan AlistarhICML 2023 · 1,240 citations
Related papers
- Skip-It? Theoretical Conditions for Layer Skipping in Vision–Language ModelsMax Hartman, Vidhata Jayaraman, Moulik Choraria, Akhil Bhimaraju et al.ICML 2026 · 1 citation
- Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models Via Adaptive Token SkippingWeili Zeng, Ziyuan Huang, Kaixiang Ji, Yichao YanICCV 2025
- Chain-of-Thought Compression Should Not Be Blind: V-Skip for Efficient Multimodal Reasoning via Dual-Path AnchoringDongxu Zhang, Yiding Sun, Cheng Tan, Wenbiao Yan et al.ACL 2026 · 18 citations
- Prune2Drive: A Plug-and-Play Framework for Accelerating Vision-Language Models in Autonomous DrivingMinhao Xiong, Zichen Wen, Zhuangcheng Gu, Xuyang Liu et al.CVPR 2026 · 17 citations
- MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert SkippingYushi Huang, Zining Wang, Zhihang Yuan, Yifu Ding et al.CVPR 2026 · 15 citations
