  Machine learning (ML) based materials discovery has emerged as one of the
most promising approaches for breakthroughs in materials science. While
heuristic knowledge based descriptors have been combined with ML algorithms to
achieve good performance, the complexity of the physicochemical mechanisms
makes it urgently needed to exploit representation learning from either
compositions or structures for building highly effective materials machine
learning models. Among these methods, the graph neural networks have shown the
best performance by its capability to learn high-level features from crystal
structures. However, all these models suffer from their inability to scale up
the models due to the over-smoothing issue of their message-passing GNN
architecture. Here we propose a novel graph attention neural network model
DeeperGATGNN with differentiable group normalization and skip-connections,
which allows to train very deep graph neural network models (e.g. 30 layers
compared to 3-9 layers in previous works). Through systematic benchmark studies
over six benchmark datasets for energy and band gap predictions, we show that
our scalable DeeperGATGNN model needs little costly hyper-parameter tuning for
different datasets and achieves the state-of-the-art prediction performances
over five properties out of six with up to 10\% improvement. Our work shows
that to deal with the high complexity of mapping the crystal materials
structures to their properties, large-scale very deep graph neural networks are
needed to achieve robust performances.
