Ted-Tok: Maintaining an Evolving Vocabulary for Lifelong Learning

Lifelong learning investigates how models adapt when exposed to a potentially infinite stream of data. Most conventional approaches focus on updating model parameters (i.e., the neural network weights) as the underlying data distribution evolves over time. However, in natural language processing, model parameters are not the only components that matter. The tokenizer, a foundational part of the system, is usually assumed to remain fixed in lifelong learning scenarios. In this work, we challenge the validity of this assumption: as language evolves, a static tokenizer fragments newly emerging lexical items, reducing compression efficiency and consequently degrading the model performance. We introduce the Temporal Drift Tokenizer (Ted-Tok), which maintains an evolving vocabulary that adapts to emerging linguistic patterns over time. This adaptivity is driven by time-weighted frequency estimators that smooth short-term fluctuations to capture persistent linguistic trends, and a principled addition-deletion strategy targeting sink tokens. Across multiple domains, Ted-Tok consistently improves compression and task performance, with gains increasing under stronger drift, underscoring the role of tokenizer adaptivity in lifelong learning. Introduction Language is not static: new concepts emerge, usage patterns shift, and meanings evolve over time. Consequently, any NLP models trained on data available at one time will become outdated in the future (Sun et al., 2020; Shi et al., 2024) . Handling this temporal drift in the data distribution is one of the core challenges in lifelong learning. To enable temporal adaptation, most prior work operates at the level of model weights -either by updating parameters during continual training or