  Self-supervised neural language models have recently found wide applications
in generative design of organic molecules and protein sequences as well as
representation learning for downstream structure classification and functional
prediction. However, most of the existing deep learning models for molecule
design usually require a big dataset and have a black-box architecture, which
makes it difficult to interpret their design logic. Here we propose Generative
Molecular Transformer (GMTransformer), a probabilistic neural network model for
generative design of molecules. Our model is built on the blank filling
language model originally developed for text processing, which has demonstrated
unique advantages in learning the "molecules grammars" with high-quality
generation, interpretability, and data efficiency. Benchmarked on the MOSES
datasets, our models achieve high novelty and Scaf compared to other baselines.
The probabilistic generation steps have the potential in tinkering molecule
design due to their capability of recommending how to modify existing molecules
with explanation, guided by the learned implicit molecule chemistry. The source
code and datasets can be accessed freely at
https://github.com/usccolumbia/GMTransformer
