Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images
Kuofeng Gao, Yang Bai, Jindong Gu, Shu-Tao Xia, Philip Torr, Zhifeng Li, Wei Liu
Abstract
Large vision-language models (VLMs) such as GPT-4 have achieved exceptional performance across various multi-modal tasks. However, the deployment of VLMs necessitates substantial energy consumption and computational resources. Once attackers maliciously induce high energy consumption and latency time (energy-latency cost) during inference of VLMs, it will exhaust computational resources. In this paper, we explore this attack surface about availability of VLMs and aim to induce high energy-latency cost during inference of VLMs. We find that high energy-latency cost during inference of VLMs can be manipulated by maximizing the length of generated sequences. To this end, we propose verbose images, with the goal of crafting an imperceptible perturbation to induce VLMs to generate long sentences during inference. Concretely, we design three loss objectives. First, a loss is proposed to delay the occurrence of end-of-sequence (EOS) token, where EOS token is a signal for VLMs to stop generating further tokens. Moreover, an uncertainty loss and a token diversity loss are proposed to increase the uncertainty over each generated token and the diversity among all tokens of the whole generated sequence, respectively, which can break output dependency at token-level and sequence-level. Furthermore, a temporal weight adjustment algorithm is proposed, which can effectively balance these losses. Extensive experiments demonstrate that our verbose images can increase the length of generated sequences by 7.87 times and 8.56 times compared to original images on MS-COCO and ImageNet datasets, which presents potential challenges for various applications. Our code is available at https://github.com/KuofengGao/Verbose_Images.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
28 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation cites The Curious Case of Neural Text Degeneration
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
- Fit the Distribution: Cross-Image/Prompt Adversarial Attacks on Multimodal Large Language Models cites Visual Instruction Tuning
- Fit the Distribution: Cross-Image/Prompt Adversarial Attacks on Multimodal Large Language Models cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Flamingo: a Visual Language Model for Few-Shot Learning cites BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Cited by top-tier papers28
- Follow-Your-Click: Open-domain Regional Image Animation via Motion PromptsYue Ma, Yingqing He, Hongfa Wang, Andong Wang et al.AAAI 2025 · 57 citations
- The VLLM Safety Paradox: Dual Ease in Jailbreak Attack and DefenseYangyang Guo, Fangkai Jiao, Liqiang Nie, Mohan KankanhalliNeurIPS 2025 · 24 citations
- Fit the Distribution: Cross-Image/Prompt Adversarial Attacks on Multimodal Large Language ModelsHai Yan, Haijian Ma, Xiaowen Cai, Daizong Liu et al.NeurIPS 2025 · 21 citations
- Make Your Home Safe: Time-aware Unsupervised User Behavior Anomaly Detection in Smart Homes via Loss-guided MaskJingyu Xiao, Zhiyao Xu, Qingsong Zou, Qing Li et al.KDD 2024 · 12 citations
- LingoLoop Attack: Trapping MLLMs via Linguistic Context and State Entrapment into Endless LoopsJiyuan Fu, Kaixun Jiang, Lingyi Hong, Jinglun Li et al.ICLR 2026 · 12 citations
Builds on23
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and GenerationJunnan Li, Dongxu Li, Caiming Xiong, Steven C. H. HoiICML 2022 · 6,549 citations
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
Related papers
- An Engorgio Prompt Makes Large Language Model Babble onJianshuo Dong, Ziyuan Zhang, Qingjie Zhang, Tianwei Zhang et al.ICLR 2025
- LoopLLM: Transferable Energy-Latency Attacks in LLMs via Repetitive GenerationXingyu Li, Xiaolei Liu, Cheng Liu, Yixiao Xu et al.AAAI 2026 · 5 citations
- Infinite Babble: Inflating 3D Vision-Language Model Inference Overhead via Adversarial Geometric PerturbationShuoyang Sun, Jiaxin Hong, Yv Zhang, Kuofeng Gao et al.ACL Findings 2026
- Less is More: Mitigating Multimodal Hallucination from an EOS Decision PerspectiveZihao Yue, Liang Zhang, Qin JinACL 2024
- Inference Optimal VLMs Need Fewer Visual Tokens and More ParametersKevin Y. Li, Sachin Goyal, João D. Semedo, J. Zico KolterICLR 2025
