EMNLP Findings2025
Local Normalization Distortion and the Thermodynamic Formalism of Decoding Strategies for Large Language Models
Tom Kempton, Stuart Burrell
Abstract
Advances in hardware and language model architecture have spurred a revolution in natural language generation. However, autoregressive models compute probability distributions over next-token choices, and sampling from these distributions, known as decoding, has received significantly less attention than other design choices. Existing decoding strategies are largely based on heuristics, resulting in methods that are difficult to apply or improve in a principled manner. We develop the theory of decoding strategies for language models by expressing popular decoding algorithms as equilibrium states in the language of ergodic theory and stating the objective functions they optimize. Using this, we analyze the effect of the local normalization step required to make probabilities sum to one in top-k, nucleus, and temperature sampling. We argue that local normalization distortion is a fundamental defect of decoding strategies and quantify the size of this distortion and its effect on mathematical proxies for the quality and diversity of generated text. This yields conclusions for the design of decoding algorithms and the detection of machine-generated text.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Closing the Curious Case of Neural Text Degeneration cites Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
- Closing the Curious Case of Neural Text Degeneration cites The Curious Case of Neural Text Degeneration
- DMAP: A Distribution Map for Text cites Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
- The Curious Case of Neural Text Degeneration cites Language GANs Falling Short
Cited by top-tier papers1
Builds on10
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- Pythia: A Suite for Analyzing Large Language Models Across Training and ScalingStella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley et al.ICML 2023 · 1,822 citations
- Language GANs Falling ShortMassimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle et al.ICLR 2020 · 236 citations
- Closing the Curious Case of Neural Text DegenerationMatthew Finlayson, John Hewitt, Alexander Koller, Swabha Swayamdipta et al.ICLR 2024 · 31 citations
Related papers
- Decoding Game: On Minimax Optimality of Heuristic Text Generation StrategiesSijin Chen, Omar Hagrass, Jason Matthew KlusowskiICLR 2025
- Automatic Detection of Generated Text is Easiest when Humans are FooledDaphne Ippolito, Daniel Duckworth, Chris Callison-Burch, Douglas EckACL 2020 · 21 citations
- Local and Global Decoding in Text GenerationDaniel Gareev, Thomas Hofmann, Ezhilmathi Krishnasamy, Tiago PimentelEMNLP Findings 2024
- Arithmetic Sampling: Parallel Diverse Decoding for Large Language ModelsLuke Vilnis, Yury Zemlyanskiy, Patrick Murray, Alexandre Tachard Passos et al.ICML 2023 · 17 citations
- A Non-monotonic Self-terminating Language ModelEugene Choi, Kyunghyun Cho, Cheolhyoung LeeICLR 2023
