  Pre-trained models are widely used in the tasks of natural language
processing nowadays. However, in the specific field of text simplification, the
research on improving pre-trained models is still blank. In this work, we
propose a continued pre-training method for text simplification. Specifically,
we propose a new masked language modeling (MLM) mechanism, which does not
randomly mask words but only masks simple words. The new mechanism can make the
model learn to generate simple words. We use a small-scale simple text dataset
for continued pre-training and employ two methods to identify simple words from
the texts. We choose BERT, a representative pre-trained model, and continue
pre-training it using our proposed method. Finally, we obtain SimpleBERT, which
surpasses BERT in both lexical simplification and sentence simplification tasks
and has achieved state-of-the-art results on multiple datasets. What's more,
SimpleBERT can replace BERT in existing simplification models without
modification.
