Towards Robust Mathematical Reasoning

Finding the right north-star metrics is highly critical for advancing mathematical reasoning capabilities of foundation models, especially given that existing evaluations are either too easy or only focusing on getting correct short answers. To address these issues, we present IMO-Bench, a suite of advanced reasoning benchmarks that specifically targets the level of the International Mathematical Olympiad (IMO), the most prestigious venue for young mathematicians. IMO-AnswerBench first tests models on 400 diverse Olympiad problems with verifiable short answers. IMO-ProofBench is the next-level evaluation for proof-writing capabilities, which includes both basic and advanced IMO problems as well as detailed grading guidelines to facilitate automatic grading. These benchmarks played a crucial role in our historic achievement of the gold-level performance at IMO 2025 with Gemini Deep Think (Luong and Lockhart, 2025). Our model achieved 80.0% on IMO-AnswerBench and 65.7% on the advanced IMO-ProofBench, surpassing the best non-Gemini models by large margins of 6.9% and 42.4% respectively. We also showed that autograders built with Gemini reasoning correlate well with human evaluations and construct IMO-GradingBench, with 1000 human gradings on proofs, to enable further progress in automatic evaluation of long-form answers. We hope that IMO-Bench will help the community towards advancing robust mathematical reasoning and release it at https://imobench . github.io. * If Correct: Correct * If Incorrect: Incorrect CRITICAL CONSTRAINT: Do not add any text, explanations, or formatting outside the <thinking> tags or the final output. Output exmaple: <thinking> 1. Golden Answer: (-∞, -4) ∪ (-4, ∞) 2. Extracted Model Answer: ∅ (the empty set) * Evaluation Categories: The expected output must be one of the following categories: 'correct', 'partial', 'almost', 'incorrect', or 'not found'. * Score Identification: The extraction is based on identifying the keyword used by the evaluator to summarize their conclusion. The criteria associated with these keywords are: * incorrect: The evaluator concluded that the solution is completely incorrect or irrelevant. * partial: The evaluator concluded that the solution is partially correct but has significant errors or omissions. * almost: The evaluator concluded that the solution is almost correct but contains minor errors or inaccuracies. * correct: The evaluator concluded that the solution is fully correct and complete. * not_found: The evaluation response does not clearly contain one of the four explicit scores listed above. * Extraction: Determine the provided score from the response and extract the category ( 'correct', 'partial', 'almost', or 'incorrect'). If a score cannot be reliably identified within the text, the output must be 'not_found'.