  Uncertainty quantification (UQ) has increasing importance in building robust
high-performance and generalizable materials property prediction models. It can
also be used in active learning to train better models by focusing on getting
new training data from uncertain regions. There are several categories of UQ
methods each considering different types of uncertainty sources. Here we
conduct a comprehensive evaluation on the UQ methods for graph neural network
based materials property prediction and evaluate how they truly reflect the
uncertainty that we want in error bound estimation or active learning. Our
experimental results over four crystal materials datasets (including formation
energy, adsorption energy, total energy, and band gap properties) show that the
popular ensemble methods for uncertainty estimation is NOT the best choice for
UQ in materials property prediction. For the convenience of the community, all
the source code and data sets can be accessed freely at
\url{https://github.com/usccolumbia/materialsUQ}.
