FG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoning

Hallucinations in large language models (LLMs) pose significant challenges in tasks requiring complex multi-step reasoning, such as mathematical problem-solving.Existing approaches primarily detect the presence of hallucinations but lack a nuanced understanding of their types and manifestations.In this paper, we first introduce a comprehensive taxonomy that categorizes the common hallucinations in mathematical reasoning tasks into six types.We then propose FG-PRM (Fine-Grained Process Reward Model), an augmented model designed to detect and mitigate hallucinations in a finegrained, step-level manner.To address the limitations of manually labeling training data, we propose an automated method for generating fine-grained hallucination data using LLMs.Our FG-PRM demonstrates superior performance across two key tasks: 1) Finegrained hallucination detection: classifying hallucination types for each reasoning step; and 2) Verification: ranking multiple LLMgenerated outputs to select the most accurate solution.Our experiments show that FG-PRM excels in fine-grained hallucination detection and substantially boosts the performance of LLMs on GSM8K and MATH benchmarks.These results highlight the benefits of finegrained supervision in enhancing the reliability and interpretability of LLM reasoning processes.Codes and datasets are available at: https://github.com/du-nlp-lab/FG-PRM.Step1: Subtract 3 from both sides: 2x + 3 -3 = 11 -3 This simplifies to: 2x = 8Step3: So, the solution to the equation is x = 4. Correct Reasoning StepsStep2: Divide both sides by 2: 2x / 2 = 8 / 2 This simplifies to: x = 4Feasibility Verification (e.g.Can Calculation Error be injected?)Hallucination Injection Yes No Question: Solve the equation 2x + 3 = 11 step by step.Correct Answer: x = 4 Yes Subtract 3 from both sides: 2x + 3 -3 = 11 -3 This simplifies to: 2x = 6 Divide both sides by 2: 2x / 2 = 8 This simplifies to: x = 5