  Within the last few years, there has been a move towards using statistical
models in conjunction with neural networks with the end goal of being able to
better answer the question, "what do our models know?". From this trend,
classical metrics such as Prediction Interval Coverage Probability (PICP) and
new metrics such as calibration error have entered the general repertoire of
model evaluation in order to gain better insight into how the uncertainty of
our model compares to reality. One important component of uncertainty modeling
is model uncertainty (epistemic uncertainty), a measurement of what the model
does and does not know. However, current evaluation techniques tends to
conflate model uncertainty with aleatoric uncertainty (irreducible error),
leading to incorrect conclusions. In this paper, using posterior predictive
checks, we show how calibration error and its variants are almost always
incorrect to use given model uncertainty, and further show how this mistake can
lead to trust in bad models and mistrust in good models. Though posterior
predictive checks has often been used for in-sample evaluation of Bayesian
models, we show it still has an important place in the modern deep learning
world.
