CVPR Findings2026
Modality-Aware Bit Allocation for Mixed-Precision Quantization of Vision-Language Models
Xi Zhang, Hanwei Zhu, Jiamang Wang, Xiaolin Wu, Weisi Lin
Abstract
We present Modality-Aware Bit Allocation (MABA), a mixed-precision post-training quantization framework for vision-language models (VLMs). MABA builds upon existing modality-aware quantization strategies and introduces an explicit bit allocation stage that distributes precision across model groups based on gradient-guided sensitivity estimates. By formulating bit assignment as a constrained optimization balancing reconstruction error and hardware budgets, MABA enables fine-grained trade-offs between accuracy and memory or latency. Applied to state-of-the-art VLMs such as LLaVA, Qwen-VL, and InternVL, MABA improves multi-modal understanding performance by up to 2.3% over strong uniform-bit baselines at equal or lower bit-width budgets. Our results highlight the effectiveness of structured bit allocation in enhancing the efficiency and adaptability of large-scale multi-modal models. Code is available at: https://github.com/xzhang9308/MABA.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
- Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
Builds on33
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations
Related papers
- Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated GradientsZiwei Xiang, Fanhu Zeng, Hongjian Fang, Rui-Qi Wang et al.CVPR 2026 · 7 citations
- MBQ: Modality-Balanced Quantization for Large Vision-Language ModelsShiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu et al.CVPR 2025
- Q-VLM: Post-training Quantization for Large Vision-Language ModelsChangyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang et al.NeurIPS 2024 · 57 citations
- Efficient Multimodal Large Language Model via Dynamic KV Cache QuantizationJiahao Fan, Chien-Ming ChenAAAI 2026
- Gated Relational Alignment via Confidence-based Distillation for Efficient VLMsYanlong Chen, Amir Habibian, Luca Benini, Yawei LiICML 2026 · 5 citations
