  With the rapid development of artificial intelligence, conversational bots
have became prevalent in mainstream E-commerce platforms, which can provide
convenient customer service timely. To satisfy the user, the conversational
bots need to understand the user's intention, detect the user's emotion, and
extract the key entities from the conversational utterances. However,
understanding dialogues is regarded as a very challenging task. Different from
common language understanding, utterances in dialogues appear alternately from
different roles and are usually organized as hierarchical structures. To
facilitate the understanding of dialogues, in this paper, we propose a novel
contextual dialogue encoder (i.e. DialogueBERT) based on the popular
pre-trained language model BERT. Five self-supervised learning pre-training
tasks are devised for learning the particularity of dialouge utterances. Four
different input embeddings are integrated to catch the relationship between
utterances, including turn embedding, role embedding, token embedding and
position embedding. DialogueBERT was pre-trained with 70 million dialogues in
real scenario, and then fine-tuned in three different downstream dialogue
understanding tasks. Experimental results show that DialogueBERT achieves
exciting results with 88.63% accuracy for intent recognition, 94.25% accuracy
for emotion recognition and 97.04% F1 score for named entity recognition, which
outperforms several strong baselines by a large margin.
