  This paper presents an ontology-aware pretrained language model (OPAL) for
end-to-end task-oriented dialogue (TOD). Unlike chit-chat dialogue models,
task-oriented dialogue models fulfill at least two task-specific modules:
dialogue state tracker (DST) and response generator (RG). The dialogue state
consists of the domain-slot-value triples, which are regarded as the user's
constraints to search the domain-related databases. The large-scale
task-oriented dialogue data with the annotated structured dialogue state
usually are inaccessible. It prevents the development of the pretrained
language model for the task-oriented dialogue. We propose a simple yet
effective pretraining method to alleviate this problem, which consists of two
pretraining phases. The first phase is to pretrain on large-scale contextual
text data, where the structured information of the text is extracted by the
information extracting tool. To bridge the gap between the pretraining method
and downstream tasks, we design two pretraining tasks: ontology-like triple
recovery and next-text generation, which simulates the DST and RG,
respectively. The second phase is to fine-tune the pretrained model on the TOD
data. The experimental results show that our proposed method achieves an
exciting boost and get competitive performance even without any TOD data on
CamRest676 and MultiWOZ benchmarks.
