  Discriminative pre-trained language models (PrLMs) can be generalized as
denoising auto-encoders that work with two procedures, ennoising and denoising.
First, an ennoising process corrupts texts with arbitrary noising functions to
construct training instances. Then, a denoising language model is trained to
restore the corrupted tokens. Existing studies have made progress by optimizing
independent strategies of either ennoising or denosing. They treat training
instances equally throughout the training process, with little attention on the
individual contribution of those instances. To model explicit signals of
instance contribution, this work proposes to estimate the complexity of
restoring the original sentences from corrupted ones in language model
pre-training. The estimations involve the corruption degree in the ennoising
data construction process and the prediction confidence in the denoising
counterpart. Experimental results on natural language understanding and reading
comprehension benchmarks show that our approach improves pre-training
efficiency, effectiveness, and robustness. Code is publicly available at
https://github.com/cooelf/InstanceReg
