  BERT (Bidirectional Encoder Representations from Transformers) has
revolutionized the field of natural language processing through its exceptional
performance on numerous tasks. Yet, the majority of researchers have mainly
concentrated on enhancements related to the model structure, such as relative
position embedding and more efficient attention mechanisms. Others have delved
into pretraining tricks associated with Masked Language Modeling, including
whole word masking. DeBERTa introduced an enhanced decoder adapted for BERT's
encoder model for pretraining, proving to be highly effective. We argue that
the design and research around enhanced masked language modeling decoders have
been underappreciated. In this paper, we propose several designs of enhanced
decoders and introduce BPDec (BERT Pretraining Decoder), a novel method for
modeling training. Typically, a pretrained BERT model is fine-tuned for
specific Natural Language Understanding (NLU) tasks. In our approach, we
utilize the original BERT model as the encoder, making only changes to the
decoder without altering the encoder. This approach does not necessitate
extensive modifications to the encoder architecture and can be seamlessly
integrated into existing fine-tuning pipelines and services, offering an
efficient and effective enhancement strategy. Compared to other methods, while
we also incur a moderate training cost for the decoder during the pretraining
process, our approach does not introduce additional training costs during the
fine-tuning phase. We test multiple enhanced decoder structures after
pretraining and evaluate their performance on the GLUE tasks and SQuAD tasks.
Our results demonstrate that BPDec, having only undergone subtle refinements to
the model structure during pretraining, significantly enhances model
performance without escalating the finetuning cost, inference time and serving
budget.
