ACL Findings2026
Dynamic Sampling that Adapts: Self-Aware Iterative Data Persistent Optimization for Mathematical Reasoning
Jun Rao, Xuebo Liu, Hexuan Deng, Zepeng Lin, Zixiong Yu, Jiansheng Wei, Xiaojun Meng, Min Zhang
Abstract
In mathematical reasoning, data selection strategies predominantly rely on static, externally defined metrics, which fail to adapt to the evolving capabilities of models during training. This misalignment limits the efficiency of Supervised Fine-Tuning and Reinforcement Learning. To bridge this gap, we introduce SAI-DPO (Self-Aware Iterative Data Persistent Optimization), a dynamic sampling framework that aligns training data with the model's intrinsic competence. SAI-DPO operationalizes two novel metrics: Knowledge Semantic Alignment for targeting domain weaknesses, and Self-Aware Difficulty, derived from pass rates and reasoning path characteristics, to gauge instance complexity relative to the model's current state. By iteratively recalibrating the data distribution based on real-time feedback, SAI-DPO dynamically aligns training samples with the model's evolving competence, ensuring the data remains strictly relevant to the model's current capability level. Extensive experiments on eight benchmarks (including AIME24 and AMC23) demonstrate that SAI-DPO outperforms static baselines at most nearly 6 points, achieving state-of-the-art efficiency with significantly less data.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Model Alignment as Prospect Theoretic Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Model Alignment as Prospect Theoretic Optimization cites Training language models to follow instructions with human feedback
- Tailoring the Training: Difficulty-Aware Learning Strategy Allocation for Large Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
- LIMA: Less Is More for Alignment cites Training language models to follow instructions with human feedback
Cited by top-tier papers3
- Iterative Closed-Loop Motion Synthesis for Scaling the Capabilities of Humanoid ControlWeisheng Xu, Qiwei Wu, Jiaxi Zhang, Jing Tan et al.CVPR 2026 · 1 citation
- Tailoring the Training: Difficulty-Aware Learning Strategy Allocation for Large Language ModelsXiaoling Zhou, Shuaiyu Zhou, Zhemg Lee, Tao Chen et al.ICML 2026
- From Past To Path: Masked History Learning for Next-Item Prediction in Generative RecommendationKaiwen Wei, Kejun He, Xiaomian Kang, Jie Zhang et al.ACL 2026
Builds on16
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Solving Quantitative Reasoning Problems with Language ModelsAitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer et al.NeurIPS 2022 · 2,039 citations
- LIMA: Less Is More for AlignmentChunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer et al.NeurIPS 2023 · 1,486 citations
- Model Alignment as Prospect Theoretic OptimizationKawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky et al.ICML 2024 · 973 citations
Related papers
- Rethinking the Sampling Criteria in Reinforcement Learning for LLM Reasoning: A Competence-Difficulty Alignment PerspectiveDeyang Kong, Qi Guo, Xiangyu Xi, Wei Wang et al.AAAI 2026 · 5 citations
- D²Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement LearningRu Zhang, Renda Li, Ziyu Ma, Weijie Qiu et al.ICML 2026
- Scale Down to Speed Up: Dynamic Data Selection for Reinforcement LearningZhuoyue Chen, Jihai Zhang, Ben Liu, Fangquan Lin et al.EMNLP Findings 2025 · 1 citation
- SPaCe: Unlocking Sample-Efficient Large Language Models Training With Self-Pace Curriculum LearningVan Dai Do, Manh Nguyen, Svetha Venkatesh, Hung LeACL Findings 2026
- Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical ReasoningHuimin Xu, Xin Mao, Feng-Lin Li, Xiaobao Wu et al.ACL Findings 2025 · 1 citation
