  Generative models have received significant attention in recent years for
materials science applications, particularly in the area of inverse design for
materials discovery. However, these models are usually assessed based on newly
generated, unverified materials, which provide a narrow evaluation of a model's
performance. Also, current efforts for inorganic materials have predominantly
focused on small crystals, even though the capability to generate large
disordered structures would significantly expand the applicability of
generative modeling. In this work, we present the Disordered Materials &
Interfaces Benchmark (Dismai-Bench), a generative model benchmark that uses
datasets of disordered alloys, interfaces, and amorphous silicon (256-264 atoms
per structure). Models are trained on each dataset independently, and evaluated
through direct structural comparisons between training and generated
structures. Benchmarking was performed on two graph diffusion models and two
(coordinate-based) U-Net diffusion models. The graph models were found to
significantly outperform the U-Net models due to the higher expressive power of
graphs. While noise in the less expressive models can assist in discovering
materials by facilitating exploration beyond the training distribution, these
models face significant challenges when confronted with more complex
structures. To further demonstrate the benefits of this benchmarking in the
development process of a generative model, we considered the case of developing
a point-cloud-based generative adversarial network (GAN) to generate low-energy
disordered interfaces. We show that the best performing architecture, CryinGAN,
outperforms the U-Net models, and is competitive against the graph models
despite its lack of invariances and weaker expressive power. This work provides
a new framework and insights to guide the development of future generative
models.
