ACL Findings2026
Learning Temporally-Aware Sample Weights for Preference Optimization
Abstract
Preference optimization is fundamental for aligning large language models. While existing methods use sample weighting, they typically rely on static functions of instantaneous model states and ignore temporal learning dynamics. We contend that a sample's value evolves throughout training, characterized by patterns such as stable convergence or noisy oscillation. We propose MetaPO, a framework that meta-learns adaptive weights using three temporal features: reward margin evolution, learning volatility, and reference deviation. Through bilevel optimization on validation data, MetaPO automatically discovers weighting strategies tailored to specific datasets. Experiments on models ranging from 7B to 70B parameters demonstrate statistically significant improvements over strong baselines, achieving gains of up to 2.4 points on AlpacaEval 2.0 and 1.6 points on Arena-Hard. Interpretability analysis confirms that temporal features drive over 70% of the weighting decisions and that the learned weights correlate strongly with sample quality.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Model Alignment as Prospect Theoretic Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Generalized Preference Optimization: A Unified Approach to Offline Alignment cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- SimPO: Simple Preference Optimization with a Reference-Free Reward cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds on11
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- SimPO: Simple Preference Optimization with a Reference-Free RewardYu Meng, Mengzhou Xia, Danqi ChenNeurIPS 2024 · 1,203 citations
- Model Alignment as Prospect Theoretic OptimizationKawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky et al.ICML 2024 · 973 citations
- ULTRAFEEDBACK: Boosting Language Models with Scaled AI FeedbackGanqu Cui, Lifan Yuan, Ning Ding, Guanming Yao et al.ICML 2024 · 286 citations
- Generalized Preference Optimization: A Unified Approach to Offline AlignmentYunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello et al.ICML 2024 · 159 citations
Related papers
- Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference OptimizationJunming Yang, Ning Xu, Biao Liu, Shiqi Qiao et al.ICLR 2026 · 3 citations
- AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference ModelsQi Liu, Jingqing Ruan, Hao Li, Haodong Zhao et al.ACL Findings 2025 · 2 citations
- Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay PerspectiveRuichen Shao, Bei Li, Gangao Liu, Yang Chen et al.ICLR 2025
- DynamicFocalPO: Adaptive Focusing Strategy for Preference OptimizationShu Zhou, Junan Chen, Rui Ling, Xin Wang et al.ACL Findings 2026
- AlphaDPO: Adaptive Reward Margin for Direct Preference OptimizationJunkang Wu, Xue Wang, Zhengyi Yang, Jiancan Wu et al.ICML 2025
