  In the real world, a learning system could receive an input that is unlike
anything it has seen during training. Unfortunately, out-of-distribution
samples can lead to unpredictable behaviour. We need to know whether any given
input belongs to the population distribution of the training/evaluation data to
prevent unpredictable behaviour in deployed systems. A recent surge of interest
in this problem has led to the development of sophisticated techniques in the
deep learning literature. However, due to the absence of a standard problem
definition or an exhaustive evaluation, it is not evident if we can rely on
these methods. What makes this problem different from a typical supervised
learning setting is that the distribution of outliers used in training may not
be the same as the distribution of outliers encountered in the application.
Classical approaches that learn inliers vs. outliers with only two datasets can
yield optimistic results. We introduce OD-test, a three-dataset evaluation
scheme as a more reliable strategy to assess progress on this problem. We
present an exhaustive evaluation of a broad set of methods from related areas
on image classification tasks. Contrary to the existing results, we show that
for realistic applications of high-dimensional images the previous techniques
have low accuracy and are not reliable in practice.
