Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, Quanquan Gu
Abstract
Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect of growing a strong LLM out of a weak one without the need for acquiring additional human-annotated data. We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from those obtained from human-annotated data. Our method progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. Theoretically, we prove that the global optimum to the training objective function of our method is achieved only when the LLM policy aligns with the target data distribution. Empirically, we evaluate our method on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our results show that SPIN can significantly improve the LLM's performance across a variety of benchmarks and even outperform models trained through direct preference optimization (DPO) supplemented with extra GPT-4 preference data. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents. Codes are available at https://github.com/uclaml/SPIN.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
237 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Model Alignment as Prospect Theoretic Optimization cites Measuring Massive Multitask Language Understanding
- Model Alignment as Prospect Theoretic Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Model Alignment as Prospect Theoretic Optimization cites Self-Rewarding Language Models
- Model Alignment as Prospect Theoretic Optimization cites Training language models to follow instructions with human feedback
- Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
- Absolute Zero: Reinforced Self-play Reasoning with Zero Data cites Self-Rewarding Language Models
- Absolute Zero: Reinforced Self-play Reasoning with Zero Data cites Training language models to follow instructions with human feedback
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- ReST-MCTS: LLM Self-Training via Process Reward Guided Tree Search cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- ReST-MCTS: LLM Self-Training via Process Reward Guided Tree Search cites Iterative Reasoning Preference Optimization
- ReST-MCTS: LLM Self-Training via Process Reward Guided Tree Search cites Self-Rewarding Language Models
- ReST-MCTS: LLM Self-Training via Process Reward Guided Tree Search cites Language Models are Few-Shot Learners
- ReST-MCTS: LLM Self-Training via Process Reward Guided Tree Search cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- ReST-MCTS: LLM Self-Training via Process Reward Guided Tree Search cites Training language models to follow instructions with human feedback
- Iterative Reasoning Preference Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Iterative Reasoning Preference Optimization cites Self-Rewarding Language Models
- Self-Rewarding Language Models cites Measuring Massive Multitask Language Understanding
- Self-Rewarding Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Self-Rewarding Language Models cites Training language models to follow instructions with human feedback
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
Cited by top-tier papers237
- Model Alignment as Prospect Theoretic OptimizationKawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky et al.ICML 2024 · 973 citations
- Self-Rewarding Language ModelsWeizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li et al.ICML 2024 · 569 citations
- ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree SearchDan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue et al.NeurIPS 2024 · 527 citations
- Absolute Zero: Reinforced Self-play Reasoning with Zero DataAndrew Zhao, Yiran Wu, Tong Wu, Quentin Xu et al.NeurIPS 2025 · 361 citations
- Iterative Reasoning Preference OptimizationRichard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho et al.NeurIPS 2024 · 287 citations
Builds on22
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
Related papers
- Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM AlignmentJiaxiang Li, Siliang Zeng, Hoi-To Wai, Chenliang Li et al.NeurIPS 2024 · 44 citations
- Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMsYibo Wang, Hai-Long Sun, Guangda Huzhang, Qingguo Chen et al.NeurIPS 2025 · 12 citations
- SPFT-SQL: Enhancing Large Language Model for Text-to-SQL Parsing by Self-Play Fine-TuningYuhao Zhang, Shaoming Duan, Jinhang Su, Chuanyi Liu et al.EMNLP Findings 2025 · 1 citation
- Self-Play Fine-tuning of Diffusion Models for Text-to-image GenerationHuizhuo Yuan, Zixiang Chen, Kaixuan Ji, Quanquan GuNeurIPS 2024 · 96 citations
- DRIFT: Difference-Aware Reinforcement Through Iterative Fine-Tuning for Language ModelWenjie Liao, Xiaohui Song, Haonan LuAAAI 2026 · 1 citation
