A Method for Assessing Inference Patterns Captured by Embedding Models in Knowledge Graphs

Various methods embed knowledge graphs with the goal of predicting missing edges. Inference patterns are the logical relationships that occur in a graph. To make proper predictions, embedding methods must capture inference patterns. There are several theoretical analyses studying pattern-capturing capabilities. Unfortunately, these analyses are challenging and many embedding methods remain unstudied. Also, they do not quantify how accurately a pattern is captured in real-world datasets. Empirical studies have been generally not consistent, and have evaluated edges in isolation. We present a model-agnostic method to empirically quantify how patterns are captured by trained embedding models. We collect the most plausible predictions to form a new graph, and use it to globally assess pattern-capturing capabilities. For a given pattern, we study positive and negative evidence, i.e., edges that the pattern deems correct and incorrect based on the partial completeness assumption. As far as we know, it is the first time negative evidence is analyzed. The assessment of a pattern measures the similarity of the positive and negative evidence between predictions and a ground truth, the original graph. Our findings indicate that several models effectively capture inference patterns for positive evidence. However, the performance is quite poor for negative evidence, which entails that models fail to learn the partial completeness assumption, even though they were trained using it. Finally, we identify new inference patterns that have not been studied before. Surprisingly, models generally achieve better performance in these new patterns that we introduce. CCS Concepts • General and reference → Evaluation; • Computing methodologies → Semantic networks; • Information systems → Data mining.