  Materials discovery driven by statistical property models is an iterative
decision process, during which an initial data collection is extended with new
data proposed by a model-informed acquisition function--with the goal to
maximize a certain "reward" over time, such as the maximum property value
discovered so far. While the materials science community achieved much progress
in developing property models that predict well on average with respect to the
training distribution, this form of in-distribution performance measurement is
not directly coupled with the discovery reward. This is because an iterative
discovery process has a shifting reward distribution that is
over-proportionally determined by the model performance for exceptional
materials. We demonstrate this problem using the example of bulk modulus
maximization among double perovskite oxides. We find that the in-distribution
predictive performance suggests random forests as superior to Gaussian process
regression, while the results are inverse in terms of the discovery rewards. We
argue that the lack of proper performance estimation methods from pre-computed
data collections is a fundamental problem for improving data-driven materials
discovery, and we propose a novel such estimator that, in contrast to na\"ive
reward estimation, successfully predicts Gaussian processes with the "expected
improvement" acquisition function as the best out of four options in our
demonstrational study for double perovskites. Importantly, it does so without
requiring the over thousand ab initio computations that were needed to confirm
this prediction.
