ReAlign: Structured Revision for Small Language Model Alignment

Aligning small language models with human preferences remains a challenging problem: weak policies often struggle to produce informative on-policy samples and exhibit unstable gradients when trained on off-policy signals from stronger models or human annotators. In this work, we introduce ReAlign, a training framework that combines the stability of on-policy learning with the guidance of reviser-assisted supervision. In ReAlign, a lightweight external reviser is first trained to improve policy-generated responses using preference-based feedback, conditioned on both the prompt and the initial output. The policy is then optimized using a hybrid approach that leverages standard on-policy preference pairs alongside reviser-enhanced pairs framed as a structured revision task. These enhanced pairs provide richer, more informative supervision, and facilitate more effective optimization. Extensive experiments on AlpacaEval-2 and Arena-Hard demonstrate that ReAlign consistently improves alignment performance for small language models and outperforms strong preference optimization baselines.