MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization

In multilingual settings, non-Latin scripts and low-resource languages are usually disadvantaged in terms of language models' utility, efficiency, and cost. Specifically, previous studies have reported multiple modeling biases that the current tokenization algorithms introduce to non-Latin script languages, the main one being over-segmentation. In this work, we propose MAGNET-multilingual adaptive gradient-based tokenization-to reduce over-segmentation via adaptive gradientbased subword tokenization. MAGNET learns to predict segment boundaries between byte tokens in a sequence via sub-modules within the model, which act as internal boundary predictors (tokenizers). Previous gradient-based tokenization methods aimed for uniform compression across sequences by integrating a single boundary predictor during training and optimizing it end-to-end through stochastic reparameterization alongside the next token prediction objective. However, this approach still results in over-segmentation for non-Latin script languages in multilingual settings. In contrast, MAGNET offers a customizable architecture where byte-level sequences are routed through language-script-specific predictors, each optimized for its respective language script. This modularity enforces equitable segmentation granularity across different language scripts compared to previous methods. Through extensive experiments, we demonstrate that in addition to reducing segmentation disparities, MAGNET also enables faster language modelling and improves downstream utility. 1 Another disparity with character-level tokenizers is that Chinese-Japanese-Korean scripts use a high number of Unicode codepoints. 2 Code and data are publicly available at https://github.com/orevaahia/ magnet-tokenization Upsampling This module converts h M l to probabilities over a byte vocabulary. This involves, first, upsampling the output of the middle block to the original resolution by duplication followed by skip 3 Nawrot et al. [31] also explore learning the segmentations using supervision from predefined word or subword boundaries. However, it is not a viable solution for all languages and does not resolve the unfairness issues. 4 P , M , and T denote representations in the middle transformer block, after pooling and at the token level. 5 The first token is defined as x0 . . . xt 1 and the last token as xt k +1 . . . xN .