Lune

EMNLP Findings2024

Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Paths

Yew Ken Chia, Guizhen Chen, Weiwen Xu, Anh Tuan Luu, Soujanya Poria, Lidong Bing

Abstract

Advanced models such as OpenAI o1 exhibit impressive problem-solving capabilities through step-by-step reasoning. However, they may still falter on more complex problems, making errors that disrupt their reasoning paths. We attribute this to the expansive solution space, where each step has the risk of diverging into mistakes. To enhance language model reasoning, we introduce a specialized training framework called Reasoning Paths Optimization (RPO), which enables learning to reason and explore from diverse paths. Our approach encourages favorable branches at each reasoning step while penalizing unfavorable ones, enhancing the model's overall problem-solving performance. Reasoning Paths Optimization does not rely on large-scale human-annotated rationales or outputs from closed-source models, making it scalable and data-efficient. We focus on multi-step reasoning tasks, such as math word problems and science-based exam questions. The experiments demonstrate that our framework significantly enhances the reasoning performance of large language models, with up to 3.1% and 4.3% improvement on GSM8K and MMLU (STEM) respectively. Our data and code can be found at https: //reasoning-paths.github.io.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • FineReason: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • D-FUSEr: Diverse Failure, Unified Success via Error-Distribution Shaping in LLM Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning cites LoRA: Low-Rank Adaptation of Large Language Models
  • FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning cites Language Models are Few-Shot Learners
  • FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning cites Training language models to follow instructions with human feedback
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. FROST: Filtering Reasoning Outliers with Attention for Efficient ReasoningHaozheng Luo, Zhuolin Jiang, Md Zahid Hasan, Yan Chen et al.ICLR 2026 · 4 citations
  2. FineReason: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle SolvingGuizhen Chen, Weiwen Xu, Hao Zhang, Hou Pong Chan et al.ACL 2025
  3. D-FUSEr: Diverse Failure, Unified Success via Error-Distribution Shaping in LLM ReasoningDavid Baek, Andrew Estornell, Yichi Zhang, Muhammad Faaiz Taufiq et al.ICML 2026

Builds on24

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  4. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  5. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations

Related papers

  1. GPO: Learning from Critical Steps to Improve LLM ReasoningJiahao Yu, Zelei Cheng, Xian Wu, Xinyu XingNeurIPS 2025 · 10 citations
  2. Learning Planning-based Reasoning by Trajectories Collection and Process Reward SynthesizingFangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F. Chen et al.EMNLP 2024 · 1 citation
  3. Search-o1: Agentic Search-Enhanced Large Reasoning ModelsXiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang et al.EMNLP 2025 · 12 citations
  4. MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMsZhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li et al.NeurIPS 2024 · 51 citations
  5. ReFT: Reasoning with Reinforced Fine-TuningLuong Quoc Trung, Xinbo Zhang, Zhanming Jie, Peng Sun et al.ACL 2024
Dusk over the sea between two cliffs drawn in fine vertical lines