  Deep neural networks have achieved outstanding performance over various
tasks, but they have a critical issue: over-confident predictions even for
completely unknown samples. Many studies have been proposed to successfully
filter out these unknown samples, but they only considered narrow and specific
tasks, referred to as misclassification detection, open-set recognition, or
out-of-distribution detection. In this work, we argue that these tasks should
be treated as fundamentally an identical problem because an ideal model should
possess detection capability for all those tasks. Therefore, we introduce the
unknown detection task, an integration of previous individual tasks, for a
rigorous examination of the detection capability of deep neural networks on a
wide spectrum of unknown samples. To this end, unified benchmark datasets on
different scales were constructed and the unknown detection capabilities of
existing popular methods were subject to comparison. We found that Deep
Ensemble consistently outperforms the other approaches in detecting unknowns;
however, all methods are only successful for a specific type of unknown. The
reproducible code and benchmark datasets are available at
https://github.com/daintlab/unknown-detection-benchmarks .
