  The use of machine learning methods for predicting the properties of
crystalline materials encounters significant challenges, primarily related to
input encoding, output versatility, and interpretability. Here, we introduce
CrystalBERT, an adaptable transformer-based framework with novel structure that
integrates space group, elemental, and unit cell information. The method's
adaptability lies not only in its ability to seamlessly combine diverse
features but also in its capability to accurately predict a wide range of
physically important properties, including topological properties,
superconducting transition temperatures, dielectric constants, and more.
CrystalBERT also provides insightful physical interpretations regarding the
features that most significantly influence the target properties. Our findings
indicate that space group and elemental information are more important for
predicting topological and superconducting properties, in contrast to some
properties that primarily depend on the unit cell information. This underscores
the intricate nature of topological and superconducting properties. By
incorporating all these features, we achieve a high accuracy of 91% in
topological classification, surpassing prior studies and identifying previously
misclassified topological materials, further demonstrating the effectiveness of
our model.
