  How do masked language models (MLMs) such as BERT learn contextual
representations? In this work, we analyze the learning dynamics of MLMs. We
find that MLMs adopt sampled embeddings as anchors to estimate and inject
contextual semantics to representations, which limits the efficiency and
effectiveness of MLMs. To address these issues, we propose TACO, a simple yet
effective representation learning approach to directly model global semantics.
TACO extracts and aligns contextual semantics hidden in contextualized
representations to encourage models to attend global semantics when generating
contextualized representations. Experiments on the GLUE benchmark show that
TACO achieves up to 5x speedup and up to 1.2 points average improvement over
existing MLMs. The code is available at https://github.com/FUZHIYI/TACO.
