  Knowledge graph completion (KGC) aims to infer missing knowledge triples
based on known facts in a knowledge graph. Current KGC research mostly follows
an entity ranking protocol, wherein the effectiveness is measured by the
predicted rank of a masked entity in a test triple. The overall performance is
then given by a micro(-average) metric over all individual answer entities. Due
to the incomplete nature of the large-scale knowledge bases, such an entity
ranking setting is likely affected by unlabelled top-ranked positive examples,
raising questions on whether the current evaluation protocol is sufficient to
guarantee a fair comparison of KGC systems. To this end, this paper presents a
systematic study on whether and how the label sparsity affects the current KGC
evaluation with the popular micro metrics. Specifically, inspired by the TREC
paradigm for large-scale information retrieval (IR) experimentation, we create
a relatively "complete" judgment set based on a sample from the popular
FB15k-237 dataset following the TREC pooling method. According to our analysis,
it comes as a surprise that switching from the original labels to our
"complete" labels results in a drastic change of system ranking of a variety of
13 popular KGC models in terms of micro metrics. Further investigation
indicates that the IR-like macro(-average) metrics are more stable and
discriminative under different settings, meanwhile, less affected by label
sparsity. Thus, for KGC evaluation, we recommend conducting TREC-style pooling
to balance between human efforts and label completeness, and reporting also the
IR-like macro metrics to reflect the ranking nature of the KGC task.
