  In open-domain dialogues, predictive uncertainties are mainly evaluated in a
domain shift setting to cope with out-of-distribution inputs. However, in
real-world conversations, there could be more extensive distributional shifted
inputs than the out-of-distribution. To evaluate this, we first propose two
methods, Unknown Word (UW) and Insufficient Context (IC), enabling gradual
distributional shifts by corruption on the dialogue dataset. We then
investigate the effect of distributional shifts on accuracy and calibration.
Our experiments show that the performance of existing uncertainty estimation
methods consistently degrades with intensifying the shift. The results suggest
that the proposed methods could be useful for evaluating the calibration of
dialogue systems under distributional shifts.
