  A medical dialogue system is essential for healthcare service as providing
primary clinical advice and diagnoses. It has been gradually adopted and
practiced in medical organizations in the form of a conversational bot, largely
due to the advancement of NLP. In recent years, the introduction of
state-of-the-art deep learning models and transfer learning techniques like
Universal Language Model Fine Tuning (ULMFiT) and Knowledge Distillation (KD)
largely contributes to the performance of NLP tasks. However, some deep neural
networks are poorly calibrated and wrongly estimate the uncertainty. Hence the
model is not trustworthy, especially in sensitive medical decision-making
systems and safety tasks. In this paper, we investigate the well-calibrated
model for ULMFiT and self-distillation (SD) in a medical dialogue system. The
calibrated ULMFiT (CULMFiT) is obtained by incorporating label smoothing (LS),
a commonly used regularization technique to achieve a well-calibrated model.
Moreover, we apply the technique to recalibrate the confidence score called
temperature scaling (TS) with KD to observe its correlation with network
calibration. To further understand the relation between SD and calibration, we
use both fixed and optimal temperatures to fine-tune the whole model. All
experiments are conducted on the consultation backpain dataset collected by
experts then further validated using a large publicly medial dialogue corpus.
We empirically show that our proposed methodologies outperform conventional
methods in terms of accuracy and robustness.
