ACL Findings2025
Forgotten Polygons: Multimodal Large Language Models are Shape-Blind
William Rudman, Michal Golovanevsky, Amir Bar, Vedant Palit, Yann LeCun, Carsten Eickhoff, Ritambhara Singh
Abstract
Despite strong performance on vision-language tasks, Multimodal Large Language Models (MLLMs) struggle with mathematical problemsolving, with both open-source and state-ofthe-art models falling short of human performance on visual-math benchmarks. To systematically examine visual-mathematical reasoning in MLLMs, we (1) evaluate their understanding of geometric primitives, (2) test multi-step reasoning, and (3) explore a potential solution to improve visual reasoning capabilities. Our findings reveal fundamental shortcomings in shape recognition, with top models achieving under 50% accuracy in identifying regular polygons. We analyze these failures through the lens of dual-process theory and show that MLLMs rely on System 1 (intuitive, memorized associations) rather than System 2 (deliberate reasoning). Consequently, MLLMs fail to count the sides of both familiar and novel shapes, suggesting they have neither learned the concept of "sides" nor effectively process visual inputs. Finally, we propose Visually Cued Chain-of-Thought (VC-CoT) prompting, which enhances multi-step mathematical reasoning by explicitly referencing visual annotations in diagrams, boosting GPT-4o's accuracy on an irregular polygon sidecounting task from 7% to 93%. Our findings suggest that System 2 reasoning in MLLMs remains an open problem, and visually-guided prompting is essential for successfully engaging visual reasoning. Code available at: https://github.com/rsinghlab/Shape-Blind * .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoning cites MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
- Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs cites Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Mechanisms of Prompt-Induced Hallucination in Vision-Language Models cites Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts
Cited by top-tier papers5
- Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMsChun-Hsiao Yeh, Chenyu Wang, Shengbang Tong, Ta Ying Cheng et al.AAAI 2026 · 35 citations
- Mechanisms of Prompt-Induced Hallucination in Vision-Language ModelsWilliam Rudman, Michal Golovanevsky, Dana Arad, Yonatan Belinkov et al.ACL 2026 · 4 citations
- Uncovering Grounding IDs: How External Cues Shape Multi-Modal BindingAmirmohammad Izadi, Hosein Hasani, Fatemeh Askari, Mobin Bagherian et al.ICML 2026 · 3 citations
- Stop Mixing Things Up! BISCUIT Teaches Vision-Language Models to Learn New Concepts from Images on the SpotJiahua Bao, Siyao Cheng, Jiaxing Du, Yuhang Jia et al.AAAI 2026
- Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual CounterfactsMichal Golovanevsky, William Rudman, Michael A. Lepori, Amir Bar et al.EMNLP 2025
Builds on13
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual ContextsPan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu et al.ICLR 2024 · 1,472 citations
- Improve Vision Language Model Chain-of-thought ReasoningRuohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang et al.ACL 2025 · 135 citations
- Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMsShengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma et al.CVPR 2024 · 111 citations
- Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical ReasoningWenwen Zhuang, Xin Huang, Xiantao Zhang, Jin ZengAAAI 2025 · 66 citations
Related papers
- Caption This, Reason That: VLMs Caught in the MiddleZihan Weng, Lucas Gomez, Taylor W. Webb, Pouya BashivanNeurIPS 2025 · 3 citations
- Primitive Vision: Improving Diagram Understanding in MLLMsShan Zhang, Aotian Chen, Yanpeng Sun, Jindong Gu et al.ICML 2025
- Whiteboard-of-Thought: Thinking Step-by-Step Across ModalitiesSachit Menon, Richard S. Zemel, Carl VondrickEMNLP 2024 · 2 citations
- Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot DoZhuoran Jin, Kejian Zhu, Hongbang Yuan, Yupu Hao et al.ACL 2026
- Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT ReasoningHai-Long Sun, Zhun Sun, Houwen Peng, Han-Jia YeACL 2025 · 23 citations
