  Large-scale pre-trained models (PTMs) such as BERT and GPT have recently
achieved great success and become a milestone in the field of artificial
intelligence (AI). Owing to sophisticated pre-training objectives and huge
model parameters, large-scale PTMs can effectively capture knowledge from
massive labeled and unlabeled data. By storing knowledge into huge parameters
and fine-tuning on specific tasks, the rich knowledge implicitly encoded in
huge parameters can benefit a variety of downstream tasks, which has been
extensively demonstrated via experimental verification and empirical analysis.
It is now the consensus of the AI community to adopt PTMs as backbone for
downstream tasks rather than learning models from scratch. In this paper, we
take a deep look into the history of pre-training, especially its special
relation with transfer learning and self-supervised learning, to reveal the
crucial position of PTMs in the AI development spectrum. Further, we
comprehensively review the latest breakthroughs of PTMs. These breakthroughs
are driven by the surge of computational power and the increasing availability
of data, towards four important directions: designing effective architectures,
utilizing rich contexts, improving computational efficiency, and conducting
interpretation and theoretical analysis. Finally, we discuss a series of open
problems and research directions of PTMs, and hope our view can inspire and
advance the future study of PTMs.
