EMNLP Findings2025
Learning Is Not A Race: Improving Retrieval in Language Models via Equal Learning
Wanqian Yang, Aahlad Manas Puli, Rajesh Ranganath
Abstract
Many applications that modern large language models (LLMs) are deployed on are retrieval tasks: the answer can be recovered from context and success is a matter of learning generalizable features from data. However, this is easier said than done. Overparametrized models trained on cross-entropy loss can overfit on noise. We argue that such overfitting is prone to happen when the model can identify mechanisms that rapidly drive down the loss of certain tokens early on in training. Fitting some tokens early reduce gradient signals in later iterations, as such, remaining tokens are more vulnerable to noise overfitting. We dub this phenomenon unequal learning and show that LLMs with longer contexts or larger embedding sizes are prone to this failure mode. In this work, we argue that learning training samples at an equal rate helps counter such biases. We highlight two mechanisms that promote equal learning: (i) loss functions that regularize uniform margins across training samples, (ii) small learning rates (e.g. by warming up) at the start of training. We demonstrate these approaches on various synthetic and natural language datasets.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on11
- Extracting Training Data from Large Language ModelsNicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski et al.USENIX Security 2021 · 2,866 citations
- The Pitfalls of Simplicity Bias in Neural NetworksHarshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain et al.NeurIPS 2020 · 503 citations
- Are Transformers universal approximators of sequence-to-sequence functions?Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi et al.ICLR 2020 · 481 citations
- How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelMichael Hanna, Ollie Liu, Alexandre VariengienNeurIPS 2023 · 251 citations
- Understanding the failure modes of out-of-distribution generalizationVaishnavh Nagarajan, Anders Andreassen, Behnam NeyshaburICLR 2021 · 205 citations
Related papers
- Forget What Matters, Keep the Rest: Selective Unlearning of Informative TokensSeunghee Koh, Sunghyun Baek, Youngdong Kim, Junmo KimACL 2026 · 1 citation
- The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text GenerationFredrik Carlsson, Fangyu Liu, Daniel Ward, Murathan Kurfali et al.ICLR 2025 · 1 citation
- Universal One-third Time Scaling in Learning Peaked DistributionsYizhou Liu, Ziming Liu, Cengiz Pehlevan, Jeff GoreICML 2026 · 6 citations
- Why Larger Language Models Do In-context Learning Differently?Zhenmei Shi, Junyi Wei, Zhuoyan Xu, Yingyu LiangICML 2024 · 54 citations
- A Closer Look at Machine Unlearning for Large Language ModelsXiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen et al.ICLR 2025
