  Obtaining accurate estimates of machine learning model uncertainties on newly
predicted data is essential for understanding the accuracy of the model and
whether its predictions can be trusted. A common approach to such uncertainty
quantification is to estimate the variance from an ensemble of models, which
are often generated by the generally applicable bootstrap method. In this work,
we demonstrate that the direct bootstrap ensemble standard deviation is not an
accurate estimate of uncertainty and propose a calibration method to
dramatically improve its accuracy. We demonstrate the effectiveness of this
calibration method for both synthetic data and physical datasets from the field
of Materials Science and Engineering. The approach is motivated by applications
in physical and biological science but is quite general and should be
applicable for uncertainty quantification in a wide range of machine learning
regression models.
