  Supervised classification methods often assume the train and test data
distributions are the same and that all classes in the test set are present in
the training set. However, deployed classifiers often require the ability to
recognize inputs from outside the training set as unknowns. This problem has
been studied under multiple paradigms including out-of-distribution detection
and open set recognition. For convolutional neural networks, there have been
two major approaches: 1) inference methods to separate knowns from unknowns and
2) feature space regularization strategies to improve model robustness to
outlier inputs. There has been little effort to explore the relationship
between the two approaches and directly compare performance on anything other
than small-scale datasets that have at most 100 categories. Using ImageNet-1K
and Places-434, we identify novel combinations of regularization and
specialized inference methods that perform best across multiple outlier
detection problems of increasing difficulty level. We found that input
perturbation and temperature scaling yield the best performance on large scale
datasets regardless of the feature space regularization strategy. Improving the
feature space by regularizing against a background class can be helpful if an
appropriate background class can be found, but this is impractical for large
scale image classification datasets.
