Lune

ACL Findings2024

GeoEval: Benchmark for Evaluating LLMs and Multi-Modal Models on Geometry Problem-Solving

Jiaxin Zhang, Zhongzhi Li, Ming-Liang Zhang, Fei Yin, Cheng-Lin Liu, Yashar Moshfeghi

Abstract

Recent advancements in large language models (LLMs) and multi-modal models (MMs) have demonstrated their remarkable capabilities in problem-solving. Yet, their proficiency in tackling geometry math problems, which necessitates an integrated understanding of both textual and visual information, has not been thoroughly evaluated. To address this gap, we introduce the GeoEval benchmark, a comprehensive collection that includes a main subset of 2,000 problems, a 750 problems subset focusing on backward reasoning, an augmented subset of 2,000 problems, and a hard subset of 300 problems. This benchmark facilitates a deeper investigation into the performance of LLMs and MMs in solving geometry math problems. Our evaluation of ten LLMs and MMs across these varied subsets reveals that the Wizard-Math model excels, achieving a 55.67% accuracy rate on the main subset but only a 6.00% accuracy on the hard subset. This highlights the critical need for testing models against datasets on which they have not been pre-trained. Additionally, our findings indicate that GPT-series models perform more effectively on problems they have rephrased, suggesting a promising method for enhancing model capabilities. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

14 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • AutoGPS: Automated Geometry Problem Solving via Multimodal Formalization and Deductive Reasoning cites MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
  • MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal Models cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers14

  1. HiddenGuard: Fine-Grained Safe Generation with Specialized Representation RouterLingrui Mei, Shenghua Liu, Yiwei Wang, Baolong Bi et al.ACL 2026 · 14 citations
  2. AutoGPS: Automated Geometry Problem Solving via Multimodal Formalization and Deductive ReasoningBowen Ping, Minnan Luo, Zhuohang Dang, Chenxi Wang et al.ICLR 2026 · 12 citations
  3. VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual ReasoningHao Yan, Xingchen Liu, Hao Wang, Zhenbiao Cao et al.ICLR 2026 · 7 citations
  4. GGBench: A Geometric Generative Reasoning Benchmark for Unified Multimodal ModelsJingxuan Wei, Caijun Jia, Xi Bai, Xinglong Xu et al.CVPR 2026 · 7 citations
  5. GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical EvaluationYuan Feng, Yue Yang, Xiaohan He, Jiatong Zhao et al.ICLR 2026 · 4 citations

Builds on11

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  3. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  4. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsLonghui Yu, Weisen Jiang, Han Shi, Jincheng Yu et al.ICLR 2024 · 637 citations
  5. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each BenchmarkOscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz et al.EMNLP Findings 2023 · 85 citations

Related papers

  1. GeoLaux: A Benchmark for Evaluating MLLMs' Geometry Performance on Long-Step Problems Requiring Auxiliary LinesYumeng Fu, Jiayin Zhu, Lingling Zhang, Wenjun Wu et al.ACL 2026 · 3 citations
  2. NoReGeo: Non-Reasoning Geometry BenchmarkIrina Abdullaeva, Anton Vasiliuk, Elizaveta Goncharova, Temurbek Rahmatullaev et al.AAAI 2026
  3. GeoGramBench: Benchmarking the Geometric Program Reasoning in Modern LLMsShixian Luo, Zhu zezhou, Yu Yuan, Yuncheng Yang et al.ICLR 2026 · 15 citations
  4. MM-MATH: Advancing Multimodal Math Evaluation with Process Evaluation and Fine-grained ClassificationKai Sun, Yushi Bai, Ji Qi, Lei Hou et al.EMNLP Findings 2024 · 4 citations
  5. Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language ModelsBofei Gao, Feifan Song, Zhe Yang, Zefan Cai et al.ICLR 2025 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines