  In real-world material research, machine learning (ML) models are usually
expected to predict and discover novel exceptional materials that deviate from
the known materials. It is thus a pressing question to provide an objective
evaluation of ML model performances in property prediction of
out-of-distribution (OOD) materials that are different from the training set
distribution. Traditional performance evaluation of materials property
prediction models through random splitting of the dataset frequently results in
artificially high performance assessments due to the inherent redundancy of
typical material datasets. Here we present a comprehensive benchmark study of
structure-based graph neural networks (GNNs) for extrapolative OOD materials
property prediction. We formulate five different categories of OOD ML problems
for three benchmark datasets from the MatBench study. Our extensive experiments
show that current state-of-the-art GNN algorithms significantly underperform
for the OOD property prediction tasks on average compared to their baselines in
the MatBench study, demonstrating a crucial generalization gap in realistic
material prediction tasks. We further examine the latent physical spaces of
these GNN models and identify the sources of CGCNN, ALIGNN, and DeeperGATGNN's
significantly more robust OOD performance than those of the current best models
in the MatBench study (coGN and coNGN), and provide insights to improve their
performance.
