  The inability of Machine Learning (ML) models to successfully extrapolate
correct predictions from out-of-distribution (OoD) samples is a major hindrance
to the application of ML in critical applications. Until the generalization
ability of ML methods is improved it is necessary to keep humans in the loop.
The need for human supervision can only be reduced if it is possible to
determining a level of confidence in predictions, which can be used to either
ask for human assistance or to abstain from making predictions. We show that
feature space distance is a meaningful measure that can provide confidence in
predictions. The distance between unseen samples and nearby training samples
proves to be correlated to the prediction error of unseen samples. Depending on
the acceptable degree of error, predictions can either be trusted or rejected
based on the distance to training samples. %Additionally, a novelty threshold
can be used to decide whether a sample is worth adding to the training set.
This enables earlier and safer deployment of models in critical applications
and is vital for deploying models under ever-changing conditions.
