WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs

Keeping large language models factually up-todate is crucial for deployment, yet costly retraining remains a challenge. Knowledge editing offers a promising alternative, but methods are only tested on small-scale or synthetic edit benchmarks. In this work, we aim to bridge research into lifelong knowledge editing to real-world edits at a practically relevant scale. We first introduce WikiBigEdit; a large-scale benchmark of realworld Wikidata edits, built to automatically extend lifelong for future-proof benchmarking. In its first instance, it includes over 500K questionanswer pairs for knowledge editing alongside a comprehensive evaluation pipeline. Finally, we use WikiBigEdit to study existing knowledge editing techniques' ability to incorporate large volumes of real-world facts and contrast their capabilities to generic modification techniques such as retrieval augmentation and continual finetuning to acquire a complete picture of the practical extent of current lifelong knowledge editing. 1 Derived from periodic changes to Wikidata knowledge graphs (Jang et al., 2022; Khodja et al., 2024) , WikiBigEdit covers a large range of factual edits and refinements. Moreover, WikiBigEdit introduces comprehensive evaluation axes going beyond standard knowl-