EMNLP Findings2024
Tokenization Falling Short: On Subword Robustness in Large Language Models
Yekun Chai, Yewei Fang, Qiwei Peng, Xuhong Li
Abstract
Language models typically tokenize raw text into sequences of subword identifiers from a predefined vocabulary, a process inherently sensitive to typographical errors, length variations, and largely oblivious to the internal structure of tokens-issues we term the curse of tokenization. In this study, we delve into these drawbacks and demonstrate that large language models (LLMs) remain susceptible to these problems. This study systematically investigates these challenges and their impact on LLMs through three critical research questions: (1) complex problem solving, (2) token structure probing, and (3) resilience to typographical variation. Our findings reveal that scaling model parameters can mitigate the issue of tokenization; however, LLMs still suffer from biases induced by typos and other text format variations. Our experiments show that subword regularization such as BPE-dropout can mitigate this issue. We release our evaluation code and data at https://github.com/FloatAI/TKEval.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
11 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Measuring Massive Multitask Language Understanding cites Language Models are Few-Shot Learners
- Apertus: Democratizing Open and Compliant LLMs for Global Language Environments cites Measuring Massive Multitask Language Understanding
- Apertus: Democratizing Open and Compliant LLMs for Global Language Environments cites TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations cites Measuring Massive Multitask Language Understanding
- TruthfulQA: Measuring How Models Mimic Human Falsehoods cites Measuring Massive Multitask Language Understanding
- HTLM: Hyper-Text Pre-Training and Prompting of Language Models cites Language Models are Few-Shot Learners
Cited by top-tier papers11
- Apertus: Democratizing Open and Compliant LLMs for Global Language EnvironmentsAlejandro Hernández-Cano, Alexander Hägele, Allen Hao Huang, Angelika Romanou et al.ACL 2026 · 51 citations
- Broken Tokens? Your Language Model can Secretly Handle Non-Canonical TokenizationsBrian Siyuan Zheng, Alisa Liu, Orevaoghene Ahia, Jonathan Hayase et al.NeurIPS 2025 · 19 citations
- The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language ModelsAdrian Cosma, Stefan Ruseti, Emilian Radoi, Mihai DascaluEMNLP 2025 · 13 citations
- TokSuite: Measuring the Impact of Tokenizer Choice on Language Model BehaviorGül Sena Altıntaş, Malikeh Ehghaghi, Brian Lester, Fengyuan Liu et al.ICML 2026 · 3 citations
- CharBench: Evaluating the Role of Tokenization in Character-Level TasksOmri Uzan, Yuval PinterAAAI 2026 · 3 citations
Builds on10
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- TruthfulQA: Measuring How Models Mimic Human FalsehoodsStephanie Lin, Jacob Hilton, Owain EvansACL 2022 · 3,228 citations
- HTLM: Hyper-Text Pre-Training and Prompting of Language ModelsArmen Aghajanyan, Dmytro Okhonko, Mike Lewis, Mandar Joshi et al.ICLR 2022 · 82 citations
- ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming LanguagesYekun Chai, Shuohuan Wang, Chao Pang, Yu Sun et al.ACL Findings 2023 · 18 citations
Related papers
- Understanding the Ability of LLMs to Handle Character-Level PerturbationAnyuan Zhuo, Xuefei Ning, Ningyuan Li, Jingyi Zhu et al.ICML 2026
- TokDrift: When LLM Speaks in Subwords but Code Speaks in GrammarYinxi Li, Yuntian Deng, Pengyu NieACL 2026 · 2 citations
- StochasTok: Improving Fine-Grained Subword Understanding in LLMsAnya Sims, Thomas Foster, T. Duy Nguyen-Hien, Klara Kaleb et al.ICLR 2026 · 8 citations
- SubTokenTest: A Practical Benchmark for Real-World Sub-token UnderstandingShuyang Hou, Yi Hu, Muhan ZhangACL 2026
- Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective DetectionYuxi Li, Yi Liu, Gelei Deng, Ying Zhang et al.FSE 2024 · 12 citations
