Lune

ACL Findings2026

Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelines

Hwanjun Song

Abstract

Much of the alignment tuning literature is organized around optimization objectives, while the construction of alignment data is often treated implicitly. In this survey, we adopt a data centric perspective and reframe alignment tuning as a pipeline design problem. We decompose alignment data construction into three interacting stages, response synthesis, preference evaluation, and preference instantiation, and use this framework to organize existing alignment methods into a unified taxonomy. Through this lens, we identify recurring design tradeoffs and failure modes observed across prior alignment methods, and distill a set of high level principles that clarify how pipeline design choices influence the resulting optimization signal. Finally, we outline open challenges for alignment data pipelines, including promptlevel alignment, agentic settings, and alignment under evolving objectives.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Self-Rewarding Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • SimPO: Simple Preference Optimization with a Reference-Free Reward cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • SimPO: Simple Preference Optimization with a Reference-Free Reward cites Self-Rewarding Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on30

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
  3. SimPO: Simple Preference Optimization with a Reference-Free RewardYu Meng, Mengzhou Xia, Danqi ChenNeurIPS 2024 · 1,203 citations
  4. Self-Rewarding Language ModelsWeizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li et al.ICML 2024 · 569 citations
  5. G-Eval: NLG Evaluation using Gpt-4 with Better Human AlignmentYang Liu, Dan Iter, Yichong Xu, Shuohang Wang et al.EMNLP 2023 · 549 citations

Related papers

  1. What Matters in Data for DPO?Yu Pan, Zhongze Cai, Huaiyang Zhong, Guanting Chen et al.NeurIPS 2025 · 13 citations
  2. Understanding Conflicts in Multi-Objective Alignment through Reward ConsistencyZhihao Xu, Yongqi Tong, Xin Zhang, Jun Zhou et al.ACL Findings 2026
  3. PopAlign: Diversifying Contrasting Patterns for a More Comprehensive AlignmentZekun Moore Wang, Shenzhi Wang, King Zhu, Jiaheng Liu et al.ACL 2025 · 2 citations
  4. LIONs: An Empirically Optimized Approach to Align Language ModelsXiao Yu, Qingyang Wu, Yu Li, Zhou YuEMNLP 2024
  5. Data Selection for LLM Alignment Using Fine-Grained PreferencesJia Zhang, Yao Liu, Chen-Xi Zhang, Yi Liu et al.ICLR 2026 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines