  This paper investigates pre-trained language models to find out which model
intrinsically carries the most informative representation for task-oriented
dialogue tasks. We approach the problem from two aspects: supervised classifier
probe and unsupervised mutual information probe. We fine-tune a feed-forward
layer as the classifier probe on top of a fixed pre-trained language model with
annotated labels in a supervised way. Meanwhile, we propose an unsupervised
mutual information probe to evaluate the mutual dependence between a real
clustering and a representation clustering. The goals of this empirical paper
are to 1) investigate probing techniques, especially from the unsupervised
mutual information aspect, 2) provide guidelines of pre-trained language model
selection for the dialogue research community, 3) find insights of pre-training
factors for dialogue application that may be the key to success.
