ACL Findings2025
Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding
Kung-Hsiang Huang, Can Qin, Haoyi Qiu, Philippe Laban, Shafiq Joty, Caiming Xiong, Chien-Sheng Wu
Abstract
Vision Language Models (VLMs) have achieved remarkable progress in multimodal tasks, yet they often struggle with visual arithmetic, seemingly simple capabilities like object counting or length comparison, which are essential for relevant complex tasks like chart understanding and geometric reasoning.In this work, we first investigate the root causes of this deficiency through a suite of probing tasks focusing on basic visual arithmetic.Our analysis reveals that while pre-trained vision encoders typically capture sufficient information, the text decoder often fails to decode it correctly for arithmetic reasoning.To address this, we propose COGALIGN, a novel post-training strategy inspired by Piaget's theory of cognitive development.COGALIGN trains VLMs to recognize invariant properties under visual transformations.We demonstrate that this approach significantly improves the performance of three diverse VLMs on our proposed probing tasks.Furthermore, COGALIGN enhances performance by an average of 4.6% on CHOCO-LATE and 2.9% on MATH-VISION, outperforming or matching supervised fine-tuning methods while requiring only 60% less training data.These results highlight the effectiveness and generalizability of COGALIGN in improving fundamental visual arithmetic capabilities and their transfer to downstream tasks. 1 Angle Comparison Question: Are these two of the same angle?GPT-4o: Yes InternVL2.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Left–Right Symmetry Breaking in CLIP-Style Vision-Language Models Trained on Synthetic Spatial-Relation Data cites Visual Instruction Tuning
- Left–Right Symmetry Breaking in CLIP-Style Vision-Language Models Trained on Synthetic Spatial-Relation Data cites Learning Transferable Visual Models From Natural Language Supervision
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- Sigmoid Loss for Language Image Pre-Training cites Learning Transferable Visual Models From Natural Language Supervision
- ReCAD: Reinforcement Learning Enhanced Parametric CAD Model Generation with Vision-Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Integrating Visual Interpretation and Linguistic Reasoning for Geometric Problem Solving cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Integrating Visual Interpretation and Linguistic Reasoning for Geometric Problem Solving cites Visual Instruction Tuning
Cited by top-tier papers5
- VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMsShmuel Berman, Jia DengNeurIPS 2025 · 7 citations
- ReCAD: Reinforcement Learning Enhanced Parametric CAD Model Generation with Vision-Language ModelsJiahao Li, Yusheng Luo, Yunzhong Lou, Xiangdong ZhouAAAI 2026 · 4 citations
- Integrating Visual Interpretation and Linguistic Reasoning for Geometric Problem SolvingZixian Guo, Ming Liu, Qilong Wang, Zhilong Ji et al.ICCV 2025 · 1 citation
- Left–Right Symmetry Breaking in CLIP-Style Vision-Language Models Trained on Synthetic Spatial-Relation DataTakaki Yamamoto, Chihiro Noguchi, Toshihiro TanizawaICML 2026
- Plug-and-Play Clarifier: A Zero-Shot Multimodal Framework for Egocentric Intent DisambiguationSicheng Yang, Yukai Huang, Weitong Cai, Shitong Sun et al.AAAI 2026
Builds on20
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
- Sigmoid Loss for Language Image Pre-TrainingXiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas BeyerICCV 2023 · 2,932 citations
Related papers
- Forgotten Polygons: Multimodal Large Language Models are Shape-BlindWilliam Rudman, Michal Golovanevsky, Amir Bar, Vedant Palit et al.ACL Findings 2025 · 9 citations
- Caption This, Reason That: VLMs Caught in the MiddleZihan Weng, Lucas Gomez, Taylor W. Webb, Pouya BashivanNeurIPS 2025 · 3 citations
- Play to Generalize: Learning to Reason Through Game PlayYunfei Xie, Yinsong Ma, Shiyi Lan, Alan L. Yuille et al.ICLR 2026 · 15 citations
- Unveiling the Compositional Ability Gap in Vision-Language Reasoning ModelTianle Li, Jihai Zhang, Yongming Rao, Yu ChengNeurIPS 2025 · 17 citations
- MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data EngineRenrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo et al.ICLR 2025
