  A well-known limitation in pretrain-finetune paradigm lies in its
inflexibility caused by the one-size-fits-all vocabulary. This potentially
weakens the effect when applying pretrained models into natural language
generation (NLG) tasks, especially for the subword distributions between
upstream and downstream tasks with significant discrepancy. Towards approaching
this problem, we extend the vanilla pretrain-finetune pipeline with an extra
embedding transfer step. Specifically, a plug-and-play embedding generator is
introduced to produce the representation of any input token, according to
pre-trained embeddings of its morphologically similar ones. Thus, embeddings of
mismatch tokens in downstream tasks can also be efficiently initialized. We
conduct experiments on a variety of NLG tasks under the pretrain-finetune
fashion. Experimental results and extensive analyses show that the proposed
strategy offers us opportunities to feel free to transfer the vocabulary,
leading to more efficient and better performed downstream NLG models.
