Lune

CVPR Findings2026

Modality-Aware Bit Allocation for Mixed-Precision Quantization of Vision-Language Models

Xi Zhang, Hanwei Zhu, Jiamang Wang, Xiaolin Wu, Weisi Lin

Abstract

We present Modality-Aware Bit Allocation (MABA), a mixed-precision post-training quantization framework for vision-language models (VLMs). MABA builds upon existing modality-aware quantization strategies and introduces an explicit bit allocation stage that distributes precision across model groups based on gradient-guided sensitivity estimates. By formulating bit assignment as a constrained optimization balancing reconstruction error and hardware budgets, MABA enables fine-grained trade-offs between accuracy and memory or latency. Applied to state-of-the-art VLMs such as LLaVA, Qwen-VL, and InternVL, MABA improves multi-modal understanding performance by up to 2.3% over strong uniform-bit baselines at equal or lower bit-width budgets. Our results highlight the effectiveness of structured bit allocation in enhancing the efficiency and adaptability of large-scale multi-modal models. Code is available at: https://github.com/xzhang9308/MABA.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
  • BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
  • MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on33

  1. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  2. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  3. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  4. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  5. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations

Related papers

  1. Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated GradientsZiwei Xiang, Fanhu Zeng, Hongjian Fang, Rui-Qi Wang et al.CVPR 2026 · 7 citations
  2. MBQ: Modality-Balanced Quantization for Large Vision-Language ModelsShiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu et al.CVPR 2025
  3. Q-VLM: Post-training Quantization for Large Vision-Language ModelsChangyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang et al.NeurIPS 2024 · 57 citations
  4. Efficient Multimodal Large Language Model via Dynamic KV Cache QuantizationJiahao Fan, Chien-Ming ChenAAAI 2026
  5. Gated Relational Alignment via Confidence-based Distillation for Efficient VLMsYanlong Chen, Amir Habibian, Luca Benini, Yawei LiICML 2026 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines