MBQ: Modality-Balanced Quantization for Large Vision-Language Models
Shiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu, Ke Hong, Xiaotao Jia, Xiuhong Li, Yaqi Yan, Pei Ran, Guohao Dai, Shengen Yan, Huazhong Yang, Yu Wang
Abstract
Vision-Language Models (VLMs) have enabled a variety of real-world applications. The large parameter size of VLMs brings large memory and computation overhead which poses significant challenges for deployment. Post-Training Quantization (PTQ) is an effective technique to reduce the memory and computation overhead. Existing PTQ methods mainly focus on large language models (LLMs), without considering the differences across other modalities. In this paper, we discover that there is a significant difference in sensitivity between language and vision tokens in large VLMs. Therefore, treating tokens from different modalities equally, as in existing PTQ methods, may over-emphasize the insensitive modalities, leading to significant accuracy loss. To deal with the above issue, we propose a simple yet effective method, Modality-Balanced Quantization (MBQ), for large VLMs. Specifically, MBQ incorporates the different sensitivities across modalities during the calibration process to minimize the reconstruction loss for better quantization parameters. Extensive experiments show that MBQ can significantly improve task accuracy by up to 4.4% and 11.6% under W3A16 and W4A8 quantization for 7B to 70B VLMs, compared to SOTA baselines. Additionally, we implement a W3A16 GPU kernel that fuses the dequantization and GEMV operators, achieving a 1.4→ speedup on LLaVAonevision-7B on the RTX 4090. The code is available at https://github.com/thu-nics/MBQ .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
12 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs cites Visual Instruction Tuning
- Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- IAG: Input-aware Backdoor Attack on VLM-based Visual Grounding cites Visual Instruction Tuning
- Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
- Visual Instruction Tuning cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
- Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients cites Visual Instruction Tuning
- Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients cites Sigmoid Loss for Language Image Pre-Training
- Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
- MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Static Quantization cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Static Quantization cites Flamingo: a Visual Language Model for Few-Shot Learning
- MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Static Quantization cites Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
Cited by top-tier papers12
- PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation ModelsTianchen Zhao, Ke Hong, Xinhao Yang, Xuefeng Xiao et al.NeurIPS 2025 · 19 citations
- IAG: Input-aware Backdoor Attack on VLM-based Visual GroundingJunxian Li, Beining Xu, Simin Chen, Jiatong Li et al.CVPR 2026 · 13 citations
- MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Static QuantizationJiangyong Yu, Sifan Zhou, Dawei Yang, Shuoyu Li et al.ACM MM 2025 · 11 citations
- Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated GradientsZiwei Xiang, Fanhu Zeng, Hongjian Fang, Rui-Qi Wang et al.CVPR 2026 · 7 citations
- Gated Relational Alignment via Confidence-based Distillation for Efficient VLMsYanlong Chen, Amir Habibian, Luca Benini, Yawei LiICML 2026 · 5 citations
Builds on22
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- Sigmoid Loss for Language Image Pre-TrainingXiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas BeyerICCV 2023 · 2,932 citations
- Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question AnsweringPan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu et al.NeurIPS 2022 · 2,727 citations
Related papers
- Modality-Aware Bit Allocation for Mixed-Precision Quantization of Vision-Language ModelsXi Zhang, Hanwei Zhu, Jiamang Wang, Xiaolin Wu, Weisi LinCVPR Findings 2026 · 1 citation
- ImpQuant: Fine-Grained Importance-Aware Quantization for Large Vision-Language ModelsJundong Zhou, Tianao Cai, Yujie Huang, Xinbing Wang et al.ICML 2026 · 2 citations
- VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language ModelsJuncan Deng, Kejie HuangCVPR 2026 · 2 citations
- Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models QuantizationChenwei Jia, Baoting Li, Xuchong Zhang, Mingzhuo Wei et al.CVPR 2026 · 3 citations
- VEQ: Modality-Adaptive Quantization for MoE Vision-Language ModelsGuangshuo Qin, Zhiteng Li, Zheng Chen, Weihang Zhang et al.ICML 2026 · 1 citation
