Lune

EMNLP Findings2022

Truncation Sampling as Language Model Desmoothing

John Hewitt, Christopher D. Manning, Percy Liang

Abstract

Long samples of text from neural language models can be of poor quality. Truncation sampling algorithms-like top-p or top-k-address this by setting some words' probabilities to zero at each step. This work provides framing for the aim of truncation, and an improved algorithm for that aim. We propose thinking of a neural language model as a mixture of a true distribution and a smoothing distribution that avoids infinite perplexity. In this light, truncation algorithms aim to perform desmoothing, estimating a subset of the support of the true distribution. Finding a good subset is crucial: we show that top-p unnecessarily truncates high-probability words, for example causing it to truncate all words but Trump for a document that starts with Donald. We introduce η-sampling, which truncates words below an entropy-dependent probability threshold. Compared to previous algorithms, η-sampling generates more plausible long English documents according to humans, is better at breaking out of repetition, and behaves more reasonably on a battery of test distributions.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

49 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers cites Language Models are Few-Shot Learners
  • MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers cites The Curious Case of Neural Text Degeneration
  • Language Models are Few-Shot Learners cites The Curious Case of Neural Text Degeneration
  • Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
  • Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites Language Models are Few-Shot Learners
  • Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads cites The Curious Case of Neural Text Degeneration
  • Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity cites The Curious Case of Neural Text Degeneration
  • Improved Natural Language Generation via Loss Truncation cites The Curious Case of Neural Text Degeneration
  • Closing the Curious Case of Neural Text Degeneration cites MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
  • Closing the Curious Case of Neural Text Degeneration cites The Curious Case of Neural Text Degeneration
  • Evaluating the World Model Implicit in a Generative Model cites The Curious Case of Neural Text Degeneration
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers49

  1. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsTianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng et al.ICML 2024 · 669 citations
  2. Evaluating the World Model Implicit in a Generative ModelKeyon Vafa, Justin Y. Chen, Ashesh Rambachan, Jon M. Kleinberg et al.NeurIPS 2024 · 166 citations
  3. Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM DiversityJiayi Zhang, Simon Yu, Derek Chong, Anthony Sicilia et al.ICML 2026 · 102 citations
  4. MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding MethodsMara Finkelstein, Markus FreitagICLR 2024 · 39 citations
  5. Closing the Curious Case of Neural Text DegenerationMatthew Finlayson, John Hewitt, Alexander Koller, Swabha Swayamdipta et al.ICLR 2024 · 31 citations

Builds on12

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  3. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence FrontiersKrishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun et al.NeurIPS 2021 · 606 citations
  4. Epistemic Neural NetworksIan Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla et al.NeurIPS 2023 · 142 citations
  5. Improved Natural Language Generation via Loss TruncationDaniel Kang, Tatsunori HashimotoACL 2020 · 72 citations

Related papers

  1. Mirostat: a Neural Text decoding Algorithm that directly controls perplexitySourya Basu, Govardana Sachitanandam Ramachandran, Nitish Shirish Keskar, Lav R. VarshneyICLR 2021 · 13 citations
  2. Neural Text Generation With Unlikelihood TrainingSean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan et al.ICLR 2020 · 683 citations
  3. Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM OutputsNguyen Nhat Minh, Andrew Baker, Clement Neo, Allen G. Roush et al.ICLR 2025
  4. Diffusion with Truncated Blocks: Fast and High-Quality Text Generation using Truncated Block GenerationYuyan Zhou, Weiyu Chen, James T. KwokACL Findings 2026
  5. Automatic Detection of Generated Text is Easiest when Humans are FooledDaphne Ippolito, Daniel Duckworth, Chris Callison-Burch, Douglas EckACL 2020 · 21 citations
Dusk over the sea between two cliffs drawn in fine vertical lines