ACL Findings2026
N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization
Xukun Zhu, Hang Yu, Peng Di, Linchao Zhu
Abstract
The success of Large Language Models in mathematical reasoning relies heavily on the generation of diverse and valid solution paths during the rollout phase. However, current rollout techniques face a fundamental tradeoff: token-level sampling often yields redundant trajectories that differ only in rephrasing, while embedding-level methods utilizing random noise frequently disrupt semantic consistency. To resolve this, we introduce N-GRPO, a novel exploration strategy integrated into the Group Relative Policy Optimization (GRPO) framework. Rather than relying on token-level sampling or native embedding-level noise, our approach leverages Semantic Neighbor Mixing. This mechanism dynamically constructs input representations by mixing the embeddings of an anchor token and its nearest semantic neighbors, thereby injecting diversity while strictly adhering to the local semantic manifold. Experimental evaluations on the DeepSeek-R1-Distill-Qwen models across different sizes show that N-GRPO not only achieves consistent improvements over strong baselines on math reasoning benchmarks but also exhibits robust generalization capabilities on out-ofdistribution tasks.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning cites DAPO: An Open-Source LLM Reinforcement Learning System at Scale
- SIM-CoT: Supervised Implicit Chain-of-Thought cites Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains
- SIM-CoT: Supervised Implicit Chain-of-Thought cites Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Space
Builds on12
- DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
- Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept SpaceZhen Zhang, Xuehai He, Weixiang Yan, Ao Shen et al.NeurIPS 2025 · 130 citations
- Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning ChainsWenhui Tan, Jiaze Li, Jianzhong Ju, Zhenbo Luo et al.NeurIPS 2025 · 103 citations
- Sample More to Think Less: Group Filtered Policy Optimization for Concise ReasoningVaishnavi Shrivastava, Ahmed Hassan Awadallah, Vidhisha Balachandran, Shivam Garg et al.ICLR 2026 · 85 citations
- SIM-CoT: Supervised Implicit Chain-of-ThoughtXilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong et al.ICLR 2026 · 58 citations
Related papers
- Smaller Models are Natural Explorers for Policy-Level Diversity in GRPOYiming Ren, Yiran Xu, Zicheng Lin, Chufan Shi et al.ICML 2026
- Lookahead Tree-Based Rollouts for Enhanced Trajectory-Level Exploration in Reinforcement Learning with Verifiable RewardsShangyu Xing, Siyuan Wang, Chenyuan Yang, Xin-Yu Dai et al.ICLR 2026 · 14 citations
- DRA-GRPO: Your GRPO Needs to Know Diverse Reasoning Paths for Mathematical ReasoningXiwen Chen, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong et al.ACL Findings 2026
- XRPO: Pushing the Limits of GRPO with Targeted Exploration and ExploitationUdbhav Bamba, Minghao Fang, Yifan Yu, Haizhong Zheng et al.ICML 2026 · 17 citations
- Geometric-Mean Policy OptimizationYuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen et al.ICLR 2026 · 104 citations
