AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling

In this paper, we introduce AceMath, a suite of frontier math models that excel in solving complex math problems, along with highly effective reward models capable of evaluating generated solutions and reliably identifying the correct ones.To develop the instructiontuned math models, we propose a supervised fine-tuning (SFT) process that first achieves competitive performance across general domains, followed by targeted fine-tuning for the math domain using a carefully curated set of prompts and synthetically generated responses.The resulting model, AceMath-72B-Instruct greatly outperforms Qwen2.5-Math-72B-Instruct,GPT-4o and Claude-3.5Sonnet.To develop math-specialized reward model, we first construct AceMath-RewardBench, a comprehensive and robust benchmark for evaluating math reward models across diverse problems and difficulty levels.After that, we present a systematic approach to build our math reward models.The resulting model, AceMath-72B-RM, consistently outperforms state-of-theart reward models.Furthermore, when combining AceMath-72B-Instruct with AceMath-72B-RM, we achieve the highest average rm@8 score across the math reasoning benchmarks.AceMath 72B-Instruct AceMath 7B-Instruct AceMath 1.5B-Instruct Qwen2.5-Math72B-Instruct Qwen2.5-Math7B-Instruct LLama3.1 405B-Instruct