  Machine learning has emerged as a novel tool for the efficient prediction of
materials properties, and claims have been made that machine-learned models for
the formation energy of compounds can approach the accuracy of Density
Functional Theory (DFT). The models tested in this work include five recently
published compositional models, a baseline model using stoichiometry alone, and
a structural model. By testing seven machine learning models for formation
energy on stability predictions using the Materials Project database of DFT
calculations for 85,014 unique chemical compositions, we show that while
formation energies can indeed be predicted well, all compositional models
perform poorly on predicting the stability of compounds, making them
considerably less useful than DFT for the discovery and design of new solids.
Most critically, in sparse chemical spaces where few stoichiometries have
stable compounds, only the structural model is capable of efficiently detecting
which materials are stable. The non-incremental improvement of structural
models compared with compositional models is noteworthy and encourages the use
of structural models for materials discovery, with the constraint that for any
new composition, the ground-state structure is not known a priori. This work
demonstrates that accurate predictions of formation energy do not imply
accurate predictions of stability, emphasizing the importance of assessing
model performance on stability predictions, for which we provide a set of
publicly available tests.
