  Reliable Confidence Estimates are hugely important for any machine learning
model to be truly useful. In this paper, we argue that any confidence estimates
based upon standard machine learning point prediction algorithms are
fundamentally flawed and under situations with a large amount of epistemic
uncertainty are likely to be untrustworthy. To address these issues, we present
MACEst, a Model Agnostic Confidence Estimator, which provides reliable and
trustworthy confidence estimates. The algorithm differs from current methods by
estimating confidence independently as a local quantity which explicitly
accounts for both aleatoric and epistemic uncertainty. This approach differs
from standard calibration methods that use a global point prediction model as a
starting point for the confidence estimate.
