  In this paper, we study trade-offs between efficiency, cost and accuracy when
pre-training Transformer encoders with different pre-training objectives. For
this purpose, we analyze features of common objectives and combine them to
create new effective pre-training approaches. Specifically, we designed light
token generators based on a straightforward statistical approach, which can
replace ELECTRA computationally heavy generators, thus highly reducing cost.
Our experiments also show that (i) there are more efficient alternatives to
BERT's MLM, and (ii) it is possible to efficiently pre-train Transformer-based
models using lighter generators without a significant drop in performance.
