Lune

EMNLP Findings2025

SeaPO: Strategic Error Amplification for Robust Preference Optimization of Large Language Models

Jun Rao, Yunjie Liao, Xuebo Liu, Zepeng Lin, Lian Lian, Dong Jin, Shengjun Cheng, Jun Yu, Min Zhang

Abstract

Existing alignment methods for preference optimization of large language models (LLMs) aim to enhance model performance by utilizing pairs of positive and negative samples. However, due to the limited capacity of models in scoring or generating responses, the quality of positive and negative samples may become similar during training, which complicates optimization for preference learning. To address this issue, we introduce SeaPO, a Strategic Error Amplification method that leverages three error types commonly occurring in LLMs to introduce specific error patterns into the model Preference Optimization. This strategy ensures that negative samples are more erroneous than positive samples and preferencebased training is employed to mitigate the occurrence of these errors, thereby enhancing model performance. Evaluations across five capability dimensions and different model scales (1.5B to 14B) demonstrate that the generated data significantly improved overall model performance, particularly in terms of truthfulness, with improvements of 5-10 percentage points observed. Further analysis reveals that task performance varies depending on the error types introduced. Injecting the most common error types improves performance in related tasks, while a mix of error types leads to a broader performance enhancement: most tasks show stable improvements, while a few tasks exhibit significant gains.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SimPO: Simple Preference Optimization with a Reference-Free Reward cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • SimPO: Simple Preference Optimization with a Reference-Free Reward cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on24

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  4. TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
  5. SimPO: Simple Preference Optimization with a Reference-Free RewardYu Meng, Mengzhou Xia, Danqi ChenNeurIPS 2024 · 1,203 citations

Related papers

  1. AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference ModelsQi Liu, Jingqing Ruan, Hao Li, Haodong Zhao et al.ACL Findings 2025 · 2 citations
  2. Self-Boosting Large Language Models with Synthetic Preference DataQingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui et al.ICLR 2025
  3. Principled Data Selection for Alignment: The Hidden Risks of Difficult ExamplesChengqian Gao, Haonan Li, Liu Liu, Zeke Xie et al.ICML 2025
  4. Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?Kexin Huang, Junkang Wu, Ziqian Chen, Xue Wang et al.ICML 2025
  5. Learning Preference Model for LLMs via Automatic Preference Data GenerationShijia Huang, Jianqiao Zhao, Yanyang Li, Liwei WangEMNLP 2023 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines