On the Generalization Gap in Self-Evolving Language Model Reasoning

Recent work suggests that large language models (LLMs) can improve through self-evolution (SE), using supervision signals generated by the model itself. In this work, we ask: under a strict closed-loop setup, where the SE algorithm has access only to an unlabeled prompt set and a base model, how close can internally generated supervision come to oracle-supervised training? We analyze four representative strategies in a unified offline self-evolution framework, including single-round verification, multi-turn revision with feedback, iterative training, and curriculum learning. Our primary experiments use Knights and Knaves (KK) logical reasoning tasks, which provide deterministic solutions, controlled difficulty levels, and a clean testbed for easy-to-hard generalization. We first show that SE consistently improves over the base model, but plateaus after excessive training compute is invested, and eventually still leaves a non trivial gap to oracle supervision. We find that multi-turn critic-revision with large models could reach strong self-evolution performance, where Gemma 12B nearly matches oracle-supervised training. Beyond KK, we also evaluate SE on real-world reasoning benchmarks, where gains are also modest. Overall, our results characterize when closed-loop SE can help, and show how internally generated supervision remains insufficient under this minimal formulation.