Rethinking Continual Knowledge Graph Embedding: Benchmarks and Analysis

Continual knowledge graph embedding (CKGE) has gained wide attention for managing dynamic knowledge graphs (KGs), which are continuously updated with new facts. Unlike traditional methods designed for static KGs, CKGE enables incremental updates to KG embeddings to accommodate new facts while retaining previously learned knowledge. Despite these advancements, current CKGE studies and benchmarks primarily focus on handling the increasing scale of data while overlooking changes in graph patterns. These changes, altering the graph structure of KGs, are referred to as pattern shifts in this paper. Pattern shifts frequently arise as new facts are added, introducing significant challenges to the stability and adaptability of CKGE methods. To address this gap, we introduce a suite of novel and challenging benchmarks, called PS-CKGE, specifically designed to evaluate CKGE methods under pattern shifts, where logic rules are utilized to capture and manage structural changes in dynamic KGs. Through these benchmarks, we comprehensively evaluate current CKGE methods in terms of their overall performance, resistance to catastrophic forgetting, and adaptability to new knowledge. The results show that pattern shifts not only exacerbate their risk of catastrophic forgetting but also impair their adaptability, usually with greater performance degradation over triples associated with more significant changes.