  Discovering novel materials can be greatly accelerated by iterative machine
learning-informed proposal of candidates---active learning. However, standard
\emph{global-scope error} metrics for model quality are not predictive of
discovery performance, and can be misleading. We introduce the notion of
\emph{Pareto shell-scope error} to help judge the suitability of a model for
proposing material candidates. Further, through synthetic cases and a
thermoelectric dataset, we probe the relation between acquisition function
fidelity and active learning performance. Results suggest novel diagnostic
tools, as well as new insights for acquisition function design.
