Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams

Large language models operating in dynamic real-world contexts often encounter knowledge that evolves continuously or emerges incrementally. To remain accurate and effective, models must adapt to newly arriving information on the fly. We introduce ONLINE ADAPTATION TO CONTINUAL KNOWLEDGE STREAMS (OAKS) to evaluate this capability, establishing a benchmark for online adaptation over streaming, continually updating knowledge. Specifically, each model is evaluated at every time interval using the same set of questions, allowing us to assess whether it can track and reason over such fine-grained knowledge dynamics across time. To support this setting, we present two datasets: OAKS-BABI and OAKS-Novel, where individual facts evolve multiple times across context chunks. These datasets include dense annotations to measure whether models track changes accurately. Evaluating 14 models with varied inference approaches, we observe significant limitations in current methodologies. Both state-of-theart models and agentic memory systems fail to adapt robustly on OAKS, demonstrating delays in state-tracking and susceptibility to distraction within streaming environments. 1