From Bytes to Subwords: Challenges of Input Representations in NLP

A first decision for any automated natural language processing system is the granularity of the input units. Traditionally, characters or words have been used, but recently, subwords have become the standard. In this paper, we investigate trends in input processing steps and discuss common shortcomings in this foundational first step of model design. We start by providing an overview of currently used tokenizers, showing that there is only minimal variety, with three highly similar designs dominating current models, and many of the tokenizers being exact duplicates. Next, we reconsider Unicode normalization strategies. Previous work has recommended applying consistent normalization; however, we argue that this removes signal and we show how this can harm performance for language classification. Finally, we take a closer look at UTF-8 character encoding, the very first layer of representation used in many language models. We argue that UTF-8 is not optimized for efficiency, nor for fairness across languages, and propose proof of concept alternatives focused on fairness and efficiency. Based on our findings, we recommend future work to 1) put more thought into subword segmentation and explore more diversity, 2) apply normalization only when beneficial 3) consider alternative character encodings for models operating on the byte-level. 1 Input Encodings in NLP For building models for the automatic processing of written natural language, one of the first decisions to make is how to represent the input. In this section, we review common input processing steps that are used in Natural Language Processing (NLP) models; we start with the smallest units (bytes, characters), and then look at subword segmentation strategies. 1 Code available on: https://bitbucket.org/ robvanderg/inputencodings/ Written natural language is commonly expressed as sequences of characters. To store texts, character encodings are used to represent characters as bytes. UTF-8 is the most commonly used character encoding today, it is used on 98.3% of the websites indexed by W3C. 2 To the best of our knowledge, all text models on HuggingFace use UTF-8. 3 It should be noted that there is a stream of work on vision-based language models that render text as an image, and use the pixels as an input (e.g. Rust et al., 2022) . These models are not dependent on UTF-8, and do not suffer from the same problems. For our main analyses, we consider these models out-of-scope, but we include a discussion of the relevance of our results for vision-based language models in Section 5. Since the 159,866 characters currently included in Unicode do not fit into a single byte, UTF-8 has a flexible byte-length (1-4). For backwards compatibility to ASCII, it represents the main Latin characters, numbers, and control characters in the first byte, but also an array of continuation bytes, which are a prefix for the higher-indexed characters. To clarify how this leads to differences in representations of text, consider the following examples: