Lune

ACL Findings2025

Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models

Hao Xiang, Bowen Yu, Hongyu Lin, Keming Lu, Yaojie Lu, Xianpei Han, Ben He, Le Sun, Jingren Zhou, Junyang Lin

Abstract

The key to effective alignment lies in highquality preference data. Recent research has focused on automated alignment, which involves developing alignment systems with minimal human intervention. However, prior research has predominantly focused on developing data generation methods, while insufficient attention has been paid to quality control mechanisms, which often produce inaccurate and unhelpful data, leading to unpredictable benefits during iterative optimization. In this paper, we present Self-Steering Optimization (SSO), an algorithm that autonomously generates highquality preference data, eliminating manual annotation requirements. SSO employs a specialized optimization objective to build a data generator from the policy model itself, which is used to produce accurate and on-policy data. We demonstrate SSO's effectiveness through comprehensive experiments on two series of models: Llama 3 and Qwen 2. Our evaluation across diverse benchmarks shows that SSO consistently outperforms baselines in human preference alignment and reward optimization. Further analysis validates SSO as a scalable framework for preference optimization, benefiting the advancement in automated alignment techniques.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Model Alignment as Prospect Theoretic Optimization cites Measuring Massive Multitask Language Understanding
  • Model Alignment as Prospect Theoretic Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Model Alignment as Prospect Theoretic Optimization cites Self-Rewarding Language Models
  • RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • \textit{S}-SPPO: Semantic-Calibrated Self-Play Preference Optimization cites Model Alignment as Prospect Theoretic Optimization
  • \textit{S}-SPPO: Semantic-Calibrated Self-Play Preference Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • \textit{S}-SPPO: Semantic-Calibrated Self-Play Preference Optimization cites Self-Rewarding Language Models
  • Self-Rewarding Language Models cites Measuring Massive Multitask Language Understanding
  • Self-Rewarding Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. S\textit{S}-SPPO: Semantic-Calibrated Self-Play Preference OptimizationXiwen Chen, Wenhui Zhu, Jingjing Wang, Peijie Qiu et al.ICML 2026

Builds on15

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. Model Alignment as Prospect Theoretic OptimizationKawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky et al.ICML 2024 · 973 citations
  4. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI FeedbackHarrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard et al.ICML 2024 · 598 citations
  5. Self-Rewarding Language ModelsWeizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li et al.ICML 2024 · 569 citations

Related papers

  1. Preference-Strength-Aware Self-Improving Alignment with Generative Preference ModelsYuanzhao Zhai, Zhuo Zhang, Cheng Yang, Kele Xu et al.SIGIR 2025
  2. Is On-Policy Data always the Best Choice for Direct Preference Optimization-Based LM Alignment?Zetian Sun, Dongfang Li, Xuhui Chen, Baotian Hu et al.ICLR 2026 · 1 citation
  3. Self-Boosting Large Language Models with Synthetic Preference DataQingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui et al.ICLR 2025
  4. Aligning Large Language Models via Fully Self-Synthetic DataShangjian Yin, Zhepei Wei, Xinyu Zhu, Wei-Lin Chen et al.ACL 2026 · 2 citations
  5. Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM GamePengyu Cheng, Yifan Yang, Jian Li, Yong Dai et al.ACL Findings 2024 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines