EMNLP Findings2024
Local and Global Decoding in Text Generation
Daniel Gareev, Thomas Hofmann, Ezhilmathi Krishnasamy, Tiago Pimentel
Abstract
Text generation, a key component in applications such as dialogue systems, relies on decoding algorithms that sample strings from a language model distribution. Traditional methods, such as top-k and top-π, apply local normalisation to the model's output distribution, which can distort it. In this paper, we investigate the effect of this distortion by introducing globallynormalised versions of these decoding methods. Additionally, we propose an independent Metropolis-Hastings algorithm to approximate sampling from globally-normalised distributions without explicitly computing them. Our empirical analysis compares the performance of local and global normalisation across two decoding algorithms (top-k and top-π) with various hyperparameters, using Pythia language models. Results show that, in most configurations, global decoding performs worse than the local decoding version of the same algorithmsdespite preserving the distribution's integrity. Our results suggest that distortion is an important feature of local decoding algorithms. lowlypalace/global-decoding
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers cites The Curious Case of Neural Text Degeneration
- Mirostat: a Neural Text decoding Algorithm that directly controls perplexity cites The Curious Case of Neural Text Degeneration
- Truncation Sampling as Language Model Desmoothing cites MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
- Truncation Sampling as Language Model Desmoothing cites Mirostat: a Neural Text decoding Algorithm that directly controls perplexity
- Truncation Sampling as Language Model Desmoothing cites The Curious Case of Neural Text Degeneration
Cited by top-tier papers1
Builds on8
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
- Pythia: A Suite for Analyzing Large Language Models Across Training and ScalingStella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley et al.ICML 2023 · 1,822 citations
- MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence FrontiersKrishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun et al.NeurIPS 2021 · 606 citations
- Truncation Sampling as Language Model DesmoothingJohn Hewitt, Christopher D. Manning, Percy LiangEMNLP Findings 2022 · 26 citations
- Mirostat: a Neural Text decoding Algorithm that directly controls perplexitySourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, Lav R. VarshneyICLR 2021 · 13 citations
Related papers
- Local Normalization Distortion and the Thermodynamic Formalism of Decoding Strategies for Large Language ModelsTom Kempton, Stuart BurrellEMNLP Findings 2025
- Decoding Game: On Minimax Optimality of Heuristic Text Generation StrategiesSijin Chen, Omar Hagrass, Jason Matthew KlusowskiICLR 2025
- Language Model Decoding as Direct Metrics OptimizationHaozhe Ji, Pei Ke, Hongning Wang, Minlie HuangICLR 2024 · 8 citations
- On the Efficacy of Sampling AdaptersClara Meister, Tiago Pimentel, Luca Malagutti, Ethan Wilcox et al.ACL 2023 · 3 citations
- Automatic Detection of Generated Text is Easiest when Humans are FooledDaphne Ippolito, Daniel Duckworth, Chris Callison-Burch, Douglas EckACL 2020 · 21 citations
