  Detecting out-of-distribution examples is important for safety-critical
machine learning applications such as detecting novel biological phenomena and
self-driving cars. However, existing research mainly focuses on simple
small-scale settings. To set the stage for more realistic out-of-distribution
detection, we depart from small-scale settings and explore large-scale
multiclass and multi-label settings with high-resolution images and thousands
of classes. To make future work in real-world settings possible, we create new
benchmarks for three large-scale settings. To test ImageNet multiclass anomaly
detectors, we introduce the Species dataset containing over 700,000 images and
over a thousand anomalous species. We leverage ImageNet-21K to evaluate PASCAL
VOC and COCO multilabel anomaly detectors. Third, we introduce a new benchmark
for anomaly segmentation by introducing a segmentation benchmark with road
anomalies. We conduct extensive experiments in these more realistic settings
for out-of-distribution detection and find that a surprisingly simple detector
based on the maximum logit outperforms prior methods in all the large-scale
multi-class, multi-label, and segmentation tasks, establishing a simple new
baseline for future work.
