  Recently, unsupervised pre-training is gaining increasing popularity in the
realm of computational linguistics, thanks to its surprising success in
advancing natural language understanding (NLU) and the potential to effectively
exploit large-scale unlabelled corpus. However, regardless of the success in
NLU, the power of unsupervised pre-training is only partially excavated when it
comes to natural language generation (NLG). The major obstacle stems from an
idiosyncratic nature of NLG: Texts are usually generated based on certain
context, which may vary with the target applications. As a result, it is
intractable to design a universal architecture for pre-training as in NLU
scenarios. Moreover, retaining the knowledge learned from pre-training when
learning on the target task is also a non-trivial problem. This review
summarizes the recent efforts to enhance NLG systems with unsupervised
pre-training, with a special focus on the methods to catalyse the integration
of pre-trained models into downstream tasks. They are classified into
architecture-based methods and strategy-based methods, based on their way of
handling the above obstacle. Discussions are also provided to give further
insights into the relationship between these two lines of work, some
informative empirical phenomenons, as well as some possible directions where
future work can be devoted to.
