  While deep neural networks provide good performance for a range of
challenging tasks, calibration and uncertainty estimation remain major
challenges, especially under distribution shift. In this paper, we propose the
amortized conditional normalized maximum likelihood (ACNML) method as a
scalable general-purpose approach for uncertainty estimation, calibration, and
out-of-distribution robustness with deep networks. Our algorithm builds on the
conditional normalized maximum likelihood (CNML) coding scheme, which has
minimax optimal properties according to the minimum description length
principle, but is computationally intractable to evaluate exactly for all but
the simplest of model classes. We propose to use approximate Bayesian inference
technqiues to produce a tractable approximation to the CNML distribution. Our
approach can be combined with any approximate inference algorithm that provides
tractable posterior densities over model parameters. We demonstrate that ACNML
compares favorably to a number of prior techniques for uncertainty estimation
in terms of calibration on out-of-distribution inputs.
