  Since 2017, the Transformer-based models play critical roles in various
downstream Natural Language Processing tasks. However, a common limitation of
the attention mechanism utilized in Transformer Encoder is that it cannot
automatically capture the information of word order, so explicit position
embeddings are generally required to be fed into the target model. In contrast,
Transformer Decoder with the causal attention masks is naturally sensitive to
the word order. In this work, we focus on improving the position encoding
ability of BERT with the causal attention masks. Furthermore, we propose a new
pre-trained language model DecBERT and evaluate it on the GLUE benchmark.
Experimental results show that (1) the causal attention mask is effective for
BERT on the language understanding tasks; (2) our DecBERT model without
position embeddings achieve comparable performance on the GLUE benchmark; and
(3) our modification accelerates the pre-training process and DecBERT w/ PE
achieves better overall performance than the baseline systems when pre-training
with the same amount of computational resources.
