ACL Findings2026
Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines
摘要
Much of the alignment tuning literature is organized around optimization objectives, while the construction of alignment data is often treated implicitly. In this survey, we adopt a data centric perspective and reframe alignment tuning as a pipeline design problem. We decompose alignment data construction into three interacting stages, response synthesis, preference evaluation, and preference instantiation, and use this framework to organize existing alignment methods into a unified taxonomy. Through this lens, we identify recurring design tradeoffs and failure modes observed across prior alignment methods, and distill a set of high level principles that clarify how pipeline design choices influence the resulting optimization signal. Finally, we outline open challenges for alignment data pipelines, including promptlevel alignment, agentic settings, and alignment under evolving objectives.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
脉络
- Self-Rewarding Language Models 引用了 Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- SimPO: Simple Preference Optimization with a Reference-Free Reward 引用了 Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- SimPO: Simple Preference Optimization with a Reference-Free Reward 引用了 Self-Rewarding Language Models
它引用的 Paper30
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning 等NeurIPS 2023 · 10,924 次引用
- DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan 等NeurIPS 2025 · 2,828 次引用
- SimPO: Simple Preference Optimization with a Reference-Free RewardYu Meng, Mengzhou Xia, Danqi ChenNeurIPS 2024 · 1,203 次引用
- Self-Rewarding Language ModelsWeizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li 等ICML 2024 · 569 次引用
- G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentYang Liu, Dan Iter, Yichong Xu, Shuohang Wang 等EMNLP 2023 · 549 次引用
相关 Paper
- What Matters in Data for DPO?Yu Pan, Zhongze Cai, Huaiyang Zhong, Guanting Chen 等NeurIPS 2025 · 13 次引用
- Understanding Conflicts in Multi-Objective Alignment through Reward ConsistencyZhihao Xu, Yongqi Tong, Xin Zhang, Jun Zhou 等ACL Findings 2026
- PopAlign: Diversifying Contrasting Patterns for a More Comprehensive AlignmentZekun Moore Wang, Shenzhi Wang, King Zhu, Jiaheng Liu 等ACL 2025 · 2 次引用
- LIONs: An Empirically Optimized Approach to Align Language ModelsXiao Yu, Qingyang Wu, Yu Li, Zhou YuEMNLP 2024
- Data Selection for LLM Alignment Using Fine-Grained PreferencesJia Zhang, Yao Liu, Chen-Xi Zhang, Yi Liu 等ICLR 2026 · 1 次引用
