  Pre-trained Language Models (PLMs) have been widely used in various natural
language processing (NLP) tasks, owing to their powerful text representations
trained on large-scale corpora. In this paper, we propose a new PLM called PERT
for natural language understanding (NLU). PERT is an auto-encoding model (like
BERT) trained with Permuted Language Model (PerLM). The formulation of the
proposed PerLM is straightforward. We permute a proportion of the input text,
and the training objective is to predict the position of the original token.
Moreover, we also apply whole word masking and N-gram masking to improve the
performance of PERT. We carried out extensive experiments on both Chinese and
English NLU benchmarks. The experimental results show that PERT can bring
improvements over various comparable baselines on some of the tasks, while
others are not. These results indicate that developing more diverse
pre-training tasks is possible instead of masked language model variants.
Several quantitative studies are carried out to better understand PERT, which
might help design PLMs in the future. Resources are available:
https://github.com/ymcui/PERT
