NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms

The performance of Large Language Models (LLMs) degrades from the temporal drift between data used for model training and newer text seen during inference. One understudied avenue of language change causing data drift is the emergence of neologisms -new word forms -over time. We create a diverse resource of recent English neologisms by using several popular collection methods. We analyze temporal drift using neologisms by comparing sentences containing new words with near-identical sentences that replace neologisms with existing substitute words. Model performance is nearly halved in machine translation when a single neologism is introduced in a sentence. Motivated by these results, we construct a benchmark to evaluate LLMs' ability to generalize to neologisms with various natural language understanding tasks and model perplexity. Models with later knowledge cutoff dates yield lower perplexities and perform better in downstream tasks. LLMs are also affected differently based on the linguistic origins of words, indicating that neologisms are complex for static LLMs to address. We will release our benchmark and code for reproducing our experiments. BART T5 GPT-J GPT-3.5 GPT-4 LLaMA-1 LLaMA-2 Pig Butchering Maskne Barbiecore Figure 1: NEO-BENCH collects neologisms from 2020-2023 for LLM evaluation. "Pig Butchering" originated as a Mandarin expression (杀猪盘). wal and Nenkova, 2022; Liu and Ritter, 2023). 042 However, as far as we are aware there has not been 043 prior work that analyzes the robustness of LLMs on 044 handling neologisms. We show that adding a neol-045 ogism to text decreases machine translation quality 046 by an average of 44% in a human evaluation ( §2),