Lune

ACL Findings2026

EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning for LLMs

Huanyu Liu, Jia Li, Yihong Dong, Chang Yu, Taozhi Chen, Lecheng Wang, Yongding Tao, Bin Gu, Ge Li

Abstract

Reinforcement learning with verifiable reward (RLVR) has become a promising paradigm for post-training large language models (LLMs) to improve their reasoning capability. However, when the rollout accuracy is low on hard problems, the reward becomes sparse, limiting learning efficiency and causing exploration bottlenecks. Existing approaches either rely on teacher models for distillation or filter out difficult problems, limiting scalability or restricting reasoning improvement through exploration.

We propose EvoCoT, a self-Evolving curriculum learning framework based on twostage Chain-of-Thought (CoT) reasoning optimization. EvoCoT constrains the exploration space by self-generating and verifying CoT trajectories, then gradually shortens CoT steps to expand the space in a controlled way. The framework enables LLMs to stably learn from initially unsolved hard problems under sparse rewards. We apply EvoCoT to multiple LLM families, including Qwen, DeepSeek, and Llama. Experiments show that EvoCoT enables LLMs to solve previously unsolved problems, improves reasoning capability without external CoT supervision, and is compatible with various RL fine-tuning methods. We release the source code and models to support future at https://github.com/gtxygyzb/EvoCoT.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
  • Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? cites STaR: Bootstrapping Reasoning With Reasoning
  • Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? cites Solving Quantitative Reasoning Problems with Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on11

  1. DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
  2. Solving Quantitative Reasoning Problems with Language ModelsAitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer et al.NeurIPS 2022 · 2,039 citations
  3. STaR: Bootstrapping Reasoning With ReasoningEric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. GoodmanNeurIPS 2022 · 1,126 citations
  4. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang et al.NeurIPS 2025 · 1,109 citations
  5. Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base ModelJingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang et al.NeurIPS 2025 · 533 citations

Related papers

  1. Rectifying LLM Thought from Lens of OptimizationJunnan Liu, Hongwei Liu, Songyang Zhang, Kai ChenICLR 2026 · 3 citations
  2. Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack EfficientlyStanley Wei, Juno KimICML 2026
  3. Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM ReasoningShubham Parashar, Shurui Gui, Xiner Li, Hongyi Ling et al.ICLR 2026 · 112 citations
  4. Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement LearningSimon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan et al.NeurIPS 2024 · 214 citations
  5. CoT-Evo: Evolutionary Distillation of Chain-of-Thought for Scientific ReasoningKehua Feng, Keyan Ding, Zhihui Zhu, Lei Liang et al.ICLR 2026 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines