  Artificial intelligence and machine learning have shown great promise in
their ability to accelerate novel materials discovery. As researchers and
domain scientists seek to unify and consolidate chemical knowledge, the case
for models with potential to generalize across different tasks within materials
science - so-called "foundation models" - grows with ambitions. This manuscript
reviews our recent progress with development of Open MatSci ML Toolkit, and
details experiments that lay the groundwork for foundation model research and
development with our framework. First, we describe and characterize a new
pretraining task that uses synthetic data generated from symmetry operations,
and reveal complex training dynamics at large scales. Using the pretrained
model, we discuss a number of use cases relevant to foundation model
development: semantic architecture of datasets, and fine-tuning for property
prediction and classification. Our key results show that for simple
applications, pretraining appears to provide worse modeling performance than
training models from random initialization. However, for more complex
instances, such as when a model is required to learn across multiple datasets
and types of targets simultaneously, the inductive bias from pretraining
provides significantly better performance. This insight will hopefully inform
subsequent efforts into creating foundation models for materials science
applications.
