  Pre-trained language models (PrLM) has been shown powerful in enhancing a
broad range of downstream tasks including various dialogue related ones.
However, PrLMs are usually trained on general plain text with common language
model (LM) training objectives, which cannot sufficiently capture dialogue
exclusive features due to the limitation of such training setting, so that
there is an immediate need to fill the gap between a specific dialogue task and
the LM task. As it is unlikely to collect huge dialogue data for
dialogue-oriented pre-training, in this paper, we propose three strategies to
simulate the conversation features on general plain text. Our proposed method
differs from existing post-training methods that it may yield a general-purpose
PrLM and does not individualize to any detailed task while keeping the
capability of learning dialogue related features including speaker awareness,
continuity and consistency. The resulted Dialog-PrLM is fine-tuned on three
public multi-turn dialogue datasets and helps achieve significant and
consistent improvement over the plain PrLMs.
