  Knowledge Graphs (KG) and associated Knowledge Graph Embedding (KGE) models
have recently begun to be explored in the context of drug discovery and have
the potential to assist in key challenges such as target identification. In the
drug discovery domain, KGs can be employed as part of a process which can
result in lab-based experiments being performed, or impact on other decisions,
incurring significant time and financial costs and most importantly, ultimately
influencing patient healthcare. For KGE models to have impact in this domain, a
better understanding of not only of performance, but also the various factors
which determine it, is required. In this study we investigate, over the course
of many thousands of experiments, the predictive performance of five KGE models
on two public drug discovery-oriented KGs. Our goal is not to focus on the best
overall model or configuration, instead we take a deeper look at how
performance can be affected by changes in the training setup, choice of
hyperparameters, model parameter initialisation seed and different splits of
the datasets. Our results highlight that these factors have significant impact
on performance and can even affect the ranking of models. Indeed these factors
should be reported along with model architectures to ensure complete
reproducibility and fair comparisons of future work, and we argue this is
critical for the acceptance of use, and impact of KGEs in a biomedical setting.
