Lune

ACL Findings2026

CPC-GRPO: Answer-Free Reinforcement Learning with Cross-Prompt Consensus Rewards

Gyunyeop Kim, Sangwoo Kang

Abstract

Reinforcement learning with verifiable rewards has improved reasoning in language models, but it typically relies on a ground-truth answer or an external verifier, which limits applicability and increases cost. We propose an answer-free training objective that derives rewards solely from the model's own probabilities by exploiting prompt paraphrases as multiple semantic views of the same intent. For each paraphrase set, we generate candidate responses, rescore each response under the other paraphrased prompts via teacher forcing, and define a cross-prompt consensus reward that serves as a practical internal training signal, favoring responses supported across views rather than those that fit only a single phrasing. We optimize this reward using a policy update with an all-pairs objective and advantage broadcasting across prompt-response pairs. The framework naturally supports prefix-level training, enabling a controllable cost-signal trade-off. Experiments on RobustAlpacaEval and out-ofdomain reasoning benchmarks (OpenBookQA, AQuA, HumanEval) show strong in-domain gains and competitive or improved average outof-domain performance over pre-trained and answer-free training baselines on LLaMA3.2-3B and Qwen3-4B, alongside analyses demonstrating reward-performance alignment and the importance of design choices such as excluding self-view scores and ensembling-based candidates. All experiment code is available at our GitHub 1 .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DAPO: An Open-Source LLM Reinforcement Learning System at Scale cites Training language models to follow instructions with human feedback
  • QLoRA: Efficient Finetuning of Quantized LLMs cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on15

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  3. DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
  4. FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri DaoICLR 2024 · 2,600 citations
  5. Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein et al.ICML 2021 · 1,843 citations

Related papers

  1. Reinforcing General Reasoning Without VerifiersXiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang et al.ICLR 2026 · 75 citations
  2. Native Reasoning Models: Training Language Models to Reason on Unverifiable DataYuanfu Wang, Zhixuan Liu, Li xiangtian, Chaochao Lu et al.ICLR 2026 · 3 citations
  3. Crossing the Reward Bridge: Expanding Reinforcement Learning with Verifiable Rewards Across Diverse DomainsYi Su, Dian Yu, Linfeng Song, Juntao Li et al.ACL 2026
  4. Verifier-Free RL for LLMs via Intrinsic Gradient-Norm RewardXuexiang Wen, Hang Yu, Linchao Zhu, Gaoang WangACL Findings 2026
  5. Self-Aligned Reward: Towards Effective and Efficient ReasonersPeixuan Han, ADIT KRISHNAN, Gerald Friedland, Jiaxuan You et al.ICLR 2026 · 10 citations
Dusk over the sea between two cliffs drawn in fine vertical lines