  Knowledge Graph (KG) completion is the problem of extending an incomplete KG
with missing facts. A key feature of Machine Learning approaches for KG
completion is their ability to learn inference patterns, so that the predicted
facts are the results of applying these patterns to the KG. Standard completion
benchmarks, however, are not well-suited for evaluating models' abilities to
learn patterns, because the training and test sets of these benchmarks are a
random split of a given KG and hence do not capture the causality of inference
patterns. We propose a novel approach for designing KG completion benchmarks
based on the following principles: there is a set of logical rules so that the
missing facts are the results of the rules' application; the training set
includes both premises matching rule antecedents and the corresponding
conclusions; the test set consists of the results of applying the rules to the
training set; the negative examples are designed to discourage the models from
learning rules not entailed by the rule set. We use our methodology to generate
several benchmarks and evaluate a wide range of existing KG completion systems.
Our results provide novel insights on the ability of existing models to induce
inference patterns from incomplete KGs.
