EMNLP Findings2023
Tokenization Consistency Matters for Generative Models on Extractive NLP Tasks
Kaiser Sun, Peng Qi, Yuhao Zhang, Lan Liu, William Yang Wang, Zhiheng Huang
Abstract
Generative models have been widely applied to solve extractive tasks, where parts of the input is extracted to form the desired output, and achieved significant success. For example, in extractive question answering (QA), generative models have constantly yielded state-of-the-art results. In this work, we study the issue of tokenization inconsistency that is commonly neglected in training these models. This issue damages the extractive nature of these tasks after the input and output are tokenized inconsistently by the tokenizer, and thus leads to performance drop as well as hallucination. We propose a simple yet effective fix to this issue and conduct a case study on extractive QA. We show that, with consistent tokenization, the model performs better in both in-domain and out-of-domain datasets, with a notable average of +1.7 F 1 gain when a BART model is trained on SQuAD and evaluated on 8 QA datasets. Further, the model converges faster, and becomes less likely to generate out-of-context answers. Our results demonstrate the need for increased scrutiny regarding how tokenization is done in extractive tasks and the benefits of consistent tokenization during training. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective cites Byte Pair Encoding is Suboptimal for Language Model Pretraining
- Charformer: Fast Character Transformers via Gradient-based Subword Tokenization cites Byte Pair Encoding is Suboptimal for Language Model Pretraining
- UnifiedQA: Crossing Format Boundaries With a Single QA System cites BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
Cited by top-tier papers4
- Watermarking Autoregressive Image GenerationNikola Jovanovic, Ismail Labiad, Tomás Soucek, Martin T. Vechev et al.NeurIPS 2025 · 21 citations
- How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve ThemDisen Liao, Freda ShiACL 2026 · 1 citation
- Addressing Tokenization Inconsistency in Steganography and Watermarking Based on Large Language ModelsRuiyi Yan, Yugo MurawakiEMNLP 2025
- Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of PerspectiveDaniel Franzen, Jan Disselhoff, David HartmannICML 2025
Builds on8
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
- UnifiedQA: Crossing Format Boundaries With a Single QA SystemDaniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal et al.EMNLP Findings 2020 · 514 citations
- Charformer: Fast Character Transformers via Gradient-based Subword TokenizationYi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Prakash Gupta et al.ICLR 2022 · 198 citations
- Byte Pair Encoding is Suboptimal for Language Model PretrainingKaj Bostrom, Greg DurrettEMNLP Findings 2020 · 180 citations
- Generate rather than Retrieve: Large Language Models are Strong Context GeneratorsWenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu et al.ICLR 2023 · 86 citations
Related papers
- Open-World Factually Consistent Question GenerationHimanshu Maheshwari, Sumit Shekhar, Apoorv Saxena, Niyati ChhayaACL Findings 2023 · 2 citations
- Token Alignment via Character Matching for Subword CompletionBen Athiwaratkun, Shiqi Wang, Mingyue Shang, Yuchen Tian et al.ACL Findings 2024 · 2 citations
- FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text modelsRakesh Chada, Pradeep NatarajanEMNLP 2021 · 36 citations
- R-Drop: Regularized Dropout for Neural NetworksXiaobo Liang, Lijun Wu, Juntao Li, Yue Wang et al.NeurIPS 2021 · 610 citations
- End-to-End Vision Tokenizer TuningWenxuan Wang, Fan Zhang, Yufeng Cui, Haiwen Diao et al.NeurIPS 2025 · 6 citations
