Task-Aware Preference Calibration for Direct Preference Optimization
Mengyang Li, Zhong Zhang, pinlong zhao
Abstract
Direct Preference Optimization (DPO) has become a predominant approach for aligning large language models with human preferences. Recent work has used perplexity differentials to identify unreliable preference labels, but these methods apply uniform calibration strategies across all samples. We observe that the reliability of perplexity signals varies substantially across task types: perplexity differentials strongly correlate with preference quality for factual tasks but provide weak signals for creative tasks where novelty is valued. Based on this observation, we propose Task-Aware Preference Calibration (TAPC), which learns task-conditioned calibration functions that adapt to the characteristics of different prompt types. TAPC employs a task encoder to extract prompt representations and learns taskspecific slope and bias parameters for mapping perplexity signals to confidence targets. Through meta-learning on a small reference dataset, TAPC discovers how to weight perplexity signals appropriately for each task category. Experiments on Llama-3-8B and Qwen2-7B demonstrate that TAPC outperforms existing methods across multiple benchmarks, with particularly large improvements on creative and open-ended tasks where uniform calibration strategies fail.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Model Alignment as Prospect Theoretic Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Model Alignment as Prospect Theoretic Optimization cites Training language models to follow instructions with human feedback
- Preference Ranking Optimization for Human Alignment cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Preference Ranking Optimization for Human Alignment cites Training language models to follow instructions with human feedback
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback cites Training language models to follow instructions with human feedback
Builds on15
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Model Alignment as Prospect Theoretic OptimizationKawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky et al.ICML 2024 · 973 citations
- Preference Ranking Optimization for Human AlignmentFeifan Song, Bowen Yu, Minghao Li, Haiyang Yu et al.AAAI 2024 · 357 citations
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI FeedbackGanqu Cui, Lifan Yuan, Ning Ding, Guanming Yao et al.ICML 2024 · 286 citations
Related papers
- Cal-DPO: Calibrated Direct Preference Optimization for Language Model AlignmentTeng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li et al.NeurIPS 2024 · 76 citations
- Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination MitigationJihao Gu, Yingyao Wang, Meng Cao, Pi Bu et al.EMNLP Findings 2025 · 2 citations
- ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference OptimizationHee Suk Yoon, Eunseop Yoon, Mark A. Hasegawa-Johnson, Sungwoong Kim et al.ICML 2025
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual FeedbackYafu Li, Xuyang Hu, Xiaoye Qu, Linjie Li et al.ICML 2025
- TUR-DPO: Topology- and Uncertainty-Aware Direct Preference OptimizationAbdulhady abas, Fatemeh Daneshfar, Seyedali Mirjalili, Mourad OussalahICML 2026
