Langevin Rollout Optimization for Modelic Reinforcement Learning

Reinforcement learning (RL) plays a central role in Large Language Model (LLM) post-training, among existing approaches, Group Relative Policy Optimization (GRPO) has become one of the most widely adopted methods, especially in RL with verifiable rewards (RLVR)-style finetuning. In GRPO, each query prompts the LLM to generate a group of rollouts with a fixed group size N . When all rollouts in a group share the same outcome-either all correct or all incorrect-the group-normalized advantages become zero, yielding no gradient signal and wasting fine-tuning compute. To mitigate this inefficiency, we introduce Adaptive Efficient Rollout Optimization (AERO). AERO addresses this by using an adaptive rollout strategy, applying selective rejection to strategically prune rollouts, and maintaining a Bayesian posterior to prevent zeroadvantage dead zones. Across three model configurations (Qwen2.5-Math-1.5B, Qwen2.5-7B, and Qwen2.5-7B-Instruct), AERO significantly improves compute efficiency without sacrificing performance. Under the same total rollout budget, AERO reduces total training compute by ∼48% while shortening wall-clock time per step by ∼45% on average. Despite the substantial reduction in compute, AERO matches or improves Pass@8 and Avg@8 over GRPO, demonstrating that it is a practical, scalable, and computeefficient strategy for RL-based LLM alignment.