Exploration-Exploitation Reshaping towards Efficient Reasoning for Large Language Models

While excelling at solving complex problems, Large Reasoning Models (LRMs) are still constrained by the overthinking issue. Most current studies rely on reward shaping in Reinforcement Learning (RL) to shorten the Chainof-Thought (CoT) of LRMs, remaining sampleinefficient and non-robust due to the absence of prioritized exploitation and guided exploration. To address these issues, we propose a novel policy optimization framework with Self-Imitation and self-Guidance MechAnisms (SIGMA), which reshapes the exploration and exploitation through two core components: (i) self-imitation exploitation, which enables the prioritized exploitation of high-value prompts and rollouts by introducing a self-imitation loss and a dynamic sampling strategy based on compression rate; (ii) self-guidance exploration, which provides a preference-aware exploration guidance through diverse and pluggable selfrewriting strategies. Experiments across various datasets indicate that our method achieves superior reasoning efficiency without compromising, and even facilitating, the overall accuracy. Furthermore, ablation studies show that the proposed mechanisms can provide flexible control interfaces for the tradeoff between the reasoning accuracy and efficiency of LRMs.