Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
Dongyoung Kim, Kimin Lee, Jinwoo Shin, Jaehyung Kim
Abstract
Aligning large language models (LLMs) with human preferences becomes a key component to obtaining state-of-the-art performance, but it yields a huge cost to construct a large human-annotated preference dataset. To tackle this problem, we propose a new framework, Spread Preference Annotation with direct preference judgment (SPA), that boosts the alignment of LLMs using only a very small amount of human-annotated preference data. Our key idea is leveraging the human prior knowledge within the small (seed) data and progressively improving the alignment of LLM, by iteratively generating the responses and learning from them with the self-annotated preference data. To be specific, we propose to derive the preference label from the logits of LLM to explicitly extract the model's inherent preference. Compared to the previous approaches using external reward models or implicit in-context learning, we observe that the proposed approach is significantly more effective. In addition, we introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data. Our experimental results demonstrate that the proposed framework significantly boosts the alignment of LLMs. For example, we achieve superior alignment performance on AlpacaEval 2.0 with only 3.3% of the ground-truth preference labels in the Ultrafeedback data compared to the cases using the entire data or state-of-the-art baselines. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
13 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Preference Optimization by Estimating the Ratio of the Data Distribution cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Preference Optimization by Estimating the Ratio of the Data Distribution cites Training language models to follow instructions with human feedback
- Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment cites Training language models to follow instructions with human feedback
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- Token-Importance Guided Direct Preference Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Token-Importance Guided Direct Preference Optimization cites Training language models to follow instructions with human feedback
- Finetuned Language Models are Zero-Shot Learners cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Finetuned Language Models are Zero-Shot Learners cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites Finetuned Language Models are Zero-Shot Learners
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
- Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generation cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generation cites Training language models to follow instructions with human feedback
- Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy cites Training language models to follow instructions with human feedback
Cited by top-tier papers13
- Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI SynergyChris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He et al.ICLR 2026 · 211 citations
- Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video GenerationXiaomeng Yang, Mengping Yang, Gong Jia, Luozheng Qin et al.ICLR 2026 · 15 citations
- Token-Importance Guided Direct Preference OptimizationNing Yang, Hai Lin, Yibo Liu, Baoliang Tian et al.ICLR 2026 · 14 citations
- Preference Optimization by Estimating the Ratio of the Data DistributionYeongmin Kim, HeeSun Bae, Byeonghu Na, Il-Chul MoonNeurIPS 2025 · 10 citations
- Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM AlignmentChu Xu, Zhixin Zhang, Tianyu Jia, Yujie JinNeurIPS 2025 · 10 citations
Builds on21
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and ConfidenceKihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang et al.NeurIPS 2020 · 5,129 citations
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
Related papers
- Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM GamePengyu Cheng, Yifan Yang, Jian Li, Yong Dai et al.ACL Findings 2024 · 6 citations
- Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt DistillationAiwei Liu, Haoping Bai, Zhiyun Lu, Xiang Kong et al.ACL 2024 · 4 citations
- Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference AlignmentWen Yang, Junhong Wu, Chen Wang, Chengqing Zong et al.ACL Findings 2025 · 2 citations
- Bootstrapping Language Models with DPO Implicit RewardsChangyu Chen, Zichen Liu, Chao Du, Tianyu Pang et al.ICLR 2025
- ComPO: Preference Alignment via Comparison OraclesPeter Chen, Xi Chen, Wotao Yin, Tianyi LinNeurIPS 2025 · 20 citations
