  The core of self-supervised learning for pre-training language models
includes pre-training task design as well as appropriate data augmentation.
Most data augmentations in language model pre-training are context-independent.
A seminal contextualized augmentation was recently proposed in ELECTRA and
achieved state-of-the-art performance by introducing an auxiliary generation
network (generator) to produce contextualized data augmentation for the
training of a main discrimination network (discriminator). This design,
however, introduces extra computation cost of the generator and a need to
adjust the relative capability between the generator and the discriminator. In
this paper, we propose a self-augmentation strategy (SAS) where a single
network is utilized for both regular pre-training and contextualized data
augmentation for the training in later epochs. Essentially, this strategy
eliminates a separate generator and uses the single network to jointly conduct
two pre-training tasks with MLM (Masked Language Modeling) and RTD (Replaced
Token Detection) heads. It avoids the challenge to search for an appropriate
size of the generator, which is critical to the performance as evidenced in
ELECTRA and its subsequent variant models. In addition, SAS is a general
strategy that can be seamlessly combined with many new techniques emerging
recently or in the future, such as the disentangled attention mechanism from
DeBERTa. Our experiments show that SAS is able to outperform ELECTRA and other
state-of-the-art models in the GLUE tasks with similar or less computation
cost.
