  The predictive capabilities of machine learning (ML) models used in materials
discovery are typically measured using simple statistics such as the
root-mean-square error (RMSE) or the coefficient of determination ($r^2$)
between ML-predicted materials property values and their known values. A
tempting assumption is that models with low error should be effective at
guiding materials discovery, and conversely, models with high error should give
poor discovery performance. However, we observe that no clear connection exists
between a "static" quantity averaged across an entire training set, such as
RMSE, and an ML property model's ability to dynamically guide the iterative
(and often extrapolative) discovery of novel materials with targeted
properties. In this work, we simulate a sequential learning (SL)-guided
materials discovery process and demonstrate a decoupling between traditional
model error metrics and model performance in guiding materials discoveries. We
show that model performance in materials discovery depends strongly on (1) the
target range within the property distribution (e.g., whether a 1st or 10th
decile material is desired); (2) the incorporation of uncertainty estimates in
the SL acquisition function; (3) whether the scientist is interested in one
discovery or many targets; and (4) how many SL iterations are allowed. To
overcome the limitations of static metrics and robustly capture SL performance,
we recommend metrics such as Discovery Yield ($DY$), a measure of how many
high-performing materials were discovered during SL, and Discovery Probability
($DP$), a measure of likelihood of discovering high-performing materials at any
point in the SL process.
