  We present a new framework AMOS that pretrains text encoders with an
Adversarial learning curriculum via a Mixture Of Signals from multiple
auxiliary generators. Following ELECTRA-style pretraining, the main encoder is
trained as a discriminator to detect replaced tokens generated by auxiliary
masked language models (MLMs). Different from ELECTRA which trains one MLM as
the generator, we jointly train multiple MLMs of different sizes to provide
training signals at various levels of difficulty. To push the discriminator to
learn better with challenging replaced tokens, we learn mixture weights over
the auxiliary MLMs' outputs to maximize the discriminator loss by
backpropagating the gradient from the discriminator via Gumbel-Softmax. For
better pretraining efficiency, we propose a way to assemble multiple MLMs into
one unified auxiliary model. AMOS outperforms ELECTRA and recent
state-of-the-art pretrained models by about 1 point on the GLUE benchmark for
BERT base-sized models.
