MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Static Quantization
Jiangyong Yu, Sifan Zhou, Dawei Yang, Shuoyu Li, Shuo Wang, Xing Hu, Chen Xu, Zukang Xu, Changyong Shu, Zhihang Yuan
Abstract
Multimodal large language models (MLLMs) have garnered widespread attention due to their ability to understand multimodal input. However, their large parameter sizes and substantial computational demands severely hinder their practical deployment and application. While quantization is an effective way to reduce model size and inference latency, its application to MLLMs remains underexplored. In this paper, we propose MQuant, a post-training quantization (PTQ) framework designed to tackle the unique challenges of multimodal large language models (MLLMs). Conventional quantization often struggles with MLLMs because of (a) high inference latency from large visual token counts, (b) distributional disparities between visual and textual tokens, and (c) extreme outliers introduced by Hadamard-based transformations. To address these issues, MQuant introduces: • Modality-Specific Static Quantization (MSQ), assigning distinct static scales for visual vs. textual tokens; • Attention-Invariant Flexible Switching (AIFS), reordering tokens to preserve casual attention while eliminating expensive token-wise scale computations; • Rotation Magnitude Suppression (RMS), mitigating weight outliers arising from online Hadamard rotations. On five mainstream MLLMs (including Qwen-VL, MiniCPM-V, CogVLM2), MQuant under W4A8 achieves near-floating-point accuracy (<1% degradation) while reducing inference latency by up to 30%, significantly outperforming existing PTQ baselines. Our MQuant effectively bridges the gap for efficient and accurate MLLMs inference in resource-constrained devices. Code will be released in https://github.com/StiphyJay/MQuant.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
9 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Point4Bit: Post Training 4-bit Quantization for Point Cloud 3D Detection cites FocusTrack: One-Stage Focus-and-Suppress Framework for 3D Point Cloud Object Tracking
- FocusTrack: One-Stage Focus-and-Suppress Framework for 3D Point Cloud Object Tracking cites Point4Bit: Post Training 4-bit Quantization for Point Cloud 3D Detection
- MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Models cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Language Models are Few-Shot Learners
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
- Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
- Flamingo: a Visual Language Model for Few-Shot Learning cites Language Models are Few-Shot Learners
- Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
- CompTrack: Information Bottleneck-Guided Low-Rank Dynamic Token Compression for Point Cloud Tracking cites Point4Bit: Post Training 4-bit Quantization for Point Cloud 3D Detection
- CompTrack: Information Bottleneck-Guided Low-Rank Dynamic Token Compression for Point Cloud Tracking cites FocusTrack: One-Stage Focus-and-Suppress Framework for 3D Point Cloud Object Tracking
Cited by top-tier papers9
- CompTrack: Information Bottleneck-Guided Low-Rank Dynamic Token Compression for Point Cloud TrackingSifan Zhou, Yichao Cao, Jiahao Nie, Yuqian Fu et al.AAAI 2026 · 9 citations
- Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware ApproachYaoxin Yang, Peng Ye, Xudong Tan, Chongjun Tu et al.CVPR 2026 · 5 citations
- MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Modelslulu hu, Xin Chen, Chen Xin, Xinhua Xu et al.CVPR 2026 · 4 citations
- FocusTrack: One-Stage Focus-and-Suppress Framework for 3D Point Cloud Object TrackingSifan Zhou, Jiahao Nie, Ziyu Zhao, Yichao Cao et al.ACM MM 2025 · 3 citations
- Point4Bit: Post Training 4-bit Quantization for Point Cloud 3D DetectionJianyu Wang, Yu Wang, Shengjie Zhao, Sifan ZhouNeurIPS 2025 · 2 citations
Builds on26
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
Related papers
- MBQ: Modality-Balanced Quantization for Large Vision-Language ModelsShiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu et al.CVPR 2025
- ImpQuant: Fine-Grained Importance-Aware Quantization for Large Vision-Language ModelsJundong Zhou, Tianao Cai, Yujie Huang, Xinbing Wang et al.ICML 2026 · 2 citations
- Modality-Aware Bit Allocation for Mixed-Precision Quantization of Vision-Language ModelsXi Zhang, Hanwei Zhu, Jiamang Wang, Xiaolin Wu, Weisi LinCVPR Findings 2026 · 1 citation
- Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated GradientsZiwei Xiang, Fanhu Zeng, Hongjian Fang, Rui-Qi Wang et al.CVPR 2026 · 7 citations
- Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient AdaptationJingjing Xie, Yuxin Zhang, Mingbao Lin, Liujuan Cao et al.ACM MM 2024 · 5 citations
