  Self-supervised neural language models have recently achieved unprecedented
success, from natural language processing to learning the languages of
biological sequences and organic molecules. These models have demonstrated
superior performance in the generation, structure classification, and
functional predictions for proteins and molecules with learned representations.
However, most of the masking-based pre-trained language models are not designed
for generative design, and their black-box nature makes it difficult to
interpret their design logic. Here we propose BLMM Crystal Transformer, a
neural network based probabilistic generative model for generative and
tinkering design of inorganic materials. Our model is built on the blank
filling language model for text generation and has demonstrated unique
advantages in learning the "materials grammars" together with high-quality
generation, interpretability, and data efficiency. It can generate chemically
valid materials compositions with as high as 89.7\% charge neutrality and
84.8\% balanced electronegativity, which are more than 4 and 8 times higher
compared to a pseudo random sampling baseline. The probabilistic generation
process of BLMM allows it to recommend tinkering operations based on learned
materials chemistry and makes it useful for materials doping. Combined with the
TCSP crysal structure prediction algorithm, We have applied our model to
discover a set of new materials as validated using DFT calculations. Our work
thus brings the unsupervised transformer language models based generative
artificial intelligence to inorganic materials. A user-friendly web app has
been developed for computational materials doping and can be accessed freely at
\url{www.materialsatlas.org/blmtinker}.
