Uncertainty Aware Learning for Language Model Alignment
Yikun Wang, Rui Zheng, Liang Ding, Qi Zhang, Dahua Lin, Dacheng Tao
Abstract
As instruction-tuned large language models (LLMs) evolve, aligning pretrained foundation models presents increasing challenges. Existing alignment strategies, which typically leverage diverse and high-quality data sources, often overlook the intrinsic uncertainty of tasks, learning all data samples equally. This may lead to suboptimal data efficiency and model performance. In response, we propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios, by introducing the sample uncertainty (elicited from more capable LLMs). We implement UAL by a simple fashion -adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Analysis shows that our UAL indeed facilitates better token clustering in the feature space, validating our hypothesis. Extensive experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning. Notably, LLMs aligned in a mixed scenario have achieved an average improvement of 10.62% on high-entropy tasks (i.e., AlpacaEval leaderboard), and 1.81% on complex low-entropy tasks (i.e., MetaMath and GSM8K).
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning cites LIMA: Less Is More for Alignment
- Code-MUE: Measuring Code LLMs’ Uncertainty through Execution-Based Semantic Interaction Graphs cites Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models cites LIMA: Less Is More for Alignment
Cited by top-tier papers4
- TruthRL: Incentivizing Truthful LLMs via Reinforcement LearningZhepei Wei, Xiao Yang, Kai Sun, Jiaqi Wang et al.ICML 2026
- Code-MUE: Measuring Code LLMs’ Uncertainty through Execution-Based Semantic Interaction GraphsXiaoning Ren, Yinxing Xue, Lei Ma, Yuheng HuangISSTA 2026
- Learning Rewrite-Invariant Reasoning with Targeted Alternation TrainingMousa Arraf, Ido Guy, Kira RadinskyICML 2026
- Let Me Teach You: Pedagogical Foundations of Feedback for Language ModelsBeatriz Borges, Niket Tandon, Tanja Käser, Antoine BosselutEMNLP 2024
Builds on11
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- LIMA: Less Is More for AlignmentChunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer et al.NeurIPS 2023 · 1,486 citations
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human FeedbackYann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang et al.NeurIPS 2023 · 948 citations
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMsMiao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li et al.ICLR 2024 · 867 citations
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language ModelsLonghui Yu, Weisen Jiang, Han Shi, Jincheng Yu et al.ICLR 2024 · 637 citations
Related papers
- Improving Model Alignment Through Collective Intelligence of Open-Source ModelsJunlin Wang, Roy Xie, Shang Zhu, Jue Wang et al.ICML 2025
- Empowering cross-lingual abilities of instruction-tuned large language models by translation-following demonstrationsLeonardo Ranaldi, Giulia Pucci, André FreitasACL Findings 2024 · 3 citations
- SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-ReflectionLiangxin Liu, Xuebo Liu, Derek F. Wong, Dongfang Li et al.NeurIPS 2024 · 49 citations
- Weak-to-Strong Search: Align Large Language Models via Searching over Small Language ModelsZhanhui Zhou, Zhixuan Liu, Jie Liu, Zhichen Dong et al.NeurIPS 2024 · 50 citations
- Instruction Tuning With Loss Over InstructionsZhengxiang Shi, Adam X. Yang, Bin Wu, Laurence Aitchison et al.NeurIPS 2024 · 55 citations
