Lune

ACL Findings2025

Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models

Yanggan Gu, Junzhuo Li, Sirui Huang, Xin Zou, Zhenghua Li, Xuming Hu

Abstract

Aligning small language models (SLMs) with human values typically involves distilling preference knowledge from large language models (LLMs). However, existing distillation methods model preference knowledge in teacher LLMs by comparing pairwise responses, overlooking the extent of difference between responses. This limitation hinders student SLMs from capturing the nuanced preferences for multiple responses. In this paper, we propose a Preference-Aligned Distillation (PAD) framework, which models teacher's preference knowledge as a probability distribution over all potential preferences, thereby providing more nuanced supervisory signals. Our insight in developing PAD is rooted in the demonstration that language models can serve as reward functions, reflecting their intrinsic preferences. Based on this, PAD comprises three key steps: (1) sampling diverse responses using high-temperature; (2) computing rewards for both teacher and student to construct their intrinsic preference; and (3) training the student's intrinsic preference distribution to align with the teacher's. Experiments on four mainstream alignment benchmarks demonstrate that PAD consistently and significantly outperforms existing approaches, achieving over 20% improvement on AlpacaEval 2 and Arena-Hard, indicating superior alignment with human preferences. Notably, on MT-Bench, using the GEMMA model family, the student trained by PAD surpasses its teacher, further validating the effectiveness of our PAD.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Preference Ranking Optimization for Human Alignment cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Probing to Refine: Reinforcement Distillation of LLM Reasoners via Explanatory Inversion cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Self-Rewarding Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • SimPO: Simple Preference Optimization with a Reference-Free Reward cites Preference Ranking Optimization for Human Alignment
  • SimPO: Simple Preference Optimization with a Reference-Free Reward cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • SimPO: Simple Preference Optimization with a Reference-Free Reward cites Self-Rewarding Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language ModelsYanggan Gu, Yuanyi Wang, Zhaoyi Yan, Yiming Zhang et al.NeurIPS 2025 · 9 citations
  2. Probing to Refine: Reinforcement Distillation of LLM Reasoners via Explanatory InversionZhen Tan, Chengshuai Zhao, Song Wang, Jundong Li et al.ICLR 2026

Builds on16

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. SimPO: Simple Preference Optimization with a Reference-Free RewardYu Meng, Mengzhou Xia, Danqi ChenNeurIPS 2024 · 1,203 citations
  3. Self-Consistency Improves Chain of Thought Reasoning in Language ModelsXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le et al.ICLR 2023 · 681 citations
  4. Self-Rewarding Language ModelsWeizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li et al.ICML 2024 · 569 citations
  5. Preference Ranking Optimization for Human AlignmentFeifan Song, Bowen Yu, Minghao Li, Haiyang Yu et al.AAAI 2024 · 357 citations

Related papers

  1. Advantage-Guided Distillation for Preference Alignment in Small Language ModelsShiping Gao, Fanqi Wan, Jiajian Guo, Xiaojun Quan et al.ICLR 2025
  2. AlignDistil: Token-Level Language Model Alignment as Adaptive Policy DistillationSongming Zhang, Xue Zhang, Tong Zhang, Bojie Hu et al.ACL 2025
  3. Beyond Pairwise: Empowering LLM Alignment With (Ranked) Choice ModelingYuxuan Tang, Yifan FengICLR 2026 · 1 citation
  4. Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay PerspectiveRuichen Shao, Bei Li, Gangao Liu, Yang Chen et al.ICLR 2025
  5. AlphaDPO: Adaptive Reward Margin for Direct Preference OptimizationJunkang Wu, Xue Wang, Zhengyi Yang, Jiancan Wu et al.ICML 2025
Dusk over the sea between two cliffs drawn in fine vertical lines