Rubric Curriculum RL: Exploiting the Generation-Verification Gap in Non-Verifiable Domains

Reinforcement learning with verifiable rewards (RLVR) on foundation models has led to significant gains in math and code generation. Extending these gains to open-ended domains remains challenging: ground-truth verification is unavailable, human annotation is expensive, and learned reward models are prone to reward hacking. We introduce Rubric Curriculum RL (RcRL), a general self-improvement method for non-verifiable tasks that requires no new data, human annotations, or stronger teacher models. RcRL exploits the generation-verification gap, where judging good outputs is easier than producing them. We combine pairwise preferences over rollouts with a curriculum over rubric criteria, yielding a more discriminative signal than absolute scoring while reducing reward hacking through a non-stationary objective. Whereas baselines plateau or collapse within a few hundred steps, RcRL preserves output entropy and keeps improving past 1000 steps. On creative writing, RcRL outputs are preferred 70.5% of the time compared to the base model, with consistent gains across multiple creative writing benchmarks and judges. On HealthBench, RcRL improves over the base model by 14.6% (full set) and 25.4% (consensus subset), outperforming all HealthBench training baselines, including instance-specific rubrics.