EMNLP Findings2025
Zipf's and Heaps' Laws for Tokens and LLM-generated Texts
Abstract
The frequency distribution of words in humanwritten texts roughly follows a simple mathematical form known as Zipf's law. Somewhat less well known is the related Heaps' law, which describes a sublinear power-law growth of vocabulary size with document size. We study the applicability of Zipf's and Heaps' laws to texts generated by Large Language Models (LLMs). We empirically show that Heaps' and Zipf's laws only hold for LLMgenerated texts in a narrow model-dependent temperature range. These temperatures have an optimal value close to t = 1 for all the base models except the large Llama models, are higher for instruction-finetuned models and do not depend on the model size or prompting. This independently confirms the recent discovery of sampling temperature dependent phase transitions in LLM-generated texts.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Finetuned Language Models are Zero-Shot Learners cites Training language models to follow instructions with human feedback
- Training language models to follow instructions with human feedback cites Finetuned Language Models are Zero-Shot Learners
Cited by top-tier papers3
- Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative AlignmentYuming Yang, Mingyoung Lai, Wanxu Zhao, Xiaoran Fan et al.ACL 2026 · 5 citations
- Spectra: Rethinking Optimizers for LLMs Under Spectral AnisotropyZhendong Huang, Hengjie Cao, Fang DONG(董方), Ruijun Huang et al.ICML 2026 · 5 citations
- Adam's Law: Textual Frequency Law on Large Language ModelsHongyuan Lu, Zixuan Li, Zefan Zhang, Bowen Cao et al.ACL 2026 · 3 citations
Builds on3
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
- The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
Related papers
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling LawsZhixuan Pan, Shaowen Wang, Pengfei Liao, Jian LiNeurIPS 2025 · 15 citations
- Zipfian WhiteningSho Yokoi, Han Bao, Hiroto Kurita, Hidetoshi ShimodairaNeurIPS 2024 · 3 citations
- Pre-trained Models Perform the Best When Token Distributions Follow Zipf's LawYanjin He, Qingkai Zeng, Meng JiangEMNLP 2025 · 1 citation
- A New Formulation of Zipf's Meaning-Frequency Law through Contextual DiversityRyo Nagata, Kumiko Tanaka-IshiiACL 2025
- Deriving Neural Scaling Laws from the Statistics of Natural LanguageFrancesco Cagnetta, Allan Raventos, Surya Ganguli, Matthieu WyartICML 2026
