  Accurate and efficient prediction of polymer properties is of great
significance in polymer design. Conventionally, expensive and time-consuming
experiments or simulations are required to evaluate polymer functions.
Recently, Transformer models, equipped with self-attention mechanisms, have
exhibited superior performance in natural language processing. However, such
methods have not been investigated in polymer sciences. Herein, we report
TransPolymer, a Transformer-based language model for polymer property
prediction. Our proposed polymer tokenizer with chemical awareness enables
learning representations from polymer sequences. Rigorous experiments on ten
polymer property prediction benchmarks demonstrate the superior performance of
TransPolymer. Moreover, we show that TransPolymer benefits from pretraining on
large unlabeled dataset via Masked Language Modeling. Experimental results
further manifest the important role of self-attention in modeling polymer
sequences. We highlight this model as a promising computational tool for
promoting rational polymer design and understanding structure-property
relationships from a data science view.
