  There is growing interest in the automated extraction of relevant information
from clinical dialogues. However, it is difficult to collect and construct
large annotated resources for clinical dialogue tasks. Recent developments in
natural language processing suggest that large-scale pre-trained language
backbones could be leveraged for such machine comprehension and information
extraction tasks. Yet, due to the gap between pre-training and downstream
clinical domains, it remains challenging to exploit the generic backbones for
domain-specific applications. Therefore, in this work, we propose a
domain-specific language pre-training, to improve performance on downstream
tasks like dialogue comprehension. Aside from the common token-level masking
pre-training method, according to the nature of human conversations and
interactive flow of multi-topic inquiry-answering dialogues, we further propose
sample generation strategies with speaker and utterance manipulation. The
conversational pre-training guides the language backbone to reconstruct the
utterances coherently based on the remaining context, thus bridging the gap
between general and specific domains. Experiments are conducted on a clinical
conversation dataset for symptom checking, where nurses inquire and discuss
symptom information with patients. We empirically show that the neural model
with our proposed approach brings improvement in the dialogue comprehension
task, and can achieve favorable results in the low resource training scenario.
