  Deep neural retrieval models have amply demonstrated their power but
estimating the reliability of their predictions remains challenging. Most
dialog response retrieval models output a single score for a response on how
relevant it is to a given question. However, the bad calibration of deep neural
network results in various uncertainty for the single score such that the
unreliable predictions always misinform user decisions. To investigate these
issues, we present an efficient calibration and uncertainty estimation
framework PG-DRR for dialog response retrieval models which adds a Gaussian
Process layer to a deterministic deep neural network and recovers conjugacy for
tractable posterior inference by P\'{o}lya-Gamma augmentation. Finally, PG-DRR
achieves the lowest empirical calibration error (ECE) in the in-domain datasets
and the distributional shift task while keeping $R_{10}@1$ and MAP performance.
