  Open-Set Classification (OSC) intends to adapt closed-set classification
models to real-world scenarios, where the classifier must correctly label
samples of known classes while rejecting previously unseen unknown samples.
Only recently, research started to investigate on algorithms that are able to
handle these unknown samples correctly. Some of these approaches address OSC by
including into the training set negative samples that a classifier learns to
reject, expecting that these data increase the robustness of the classifier on
unknown classes. Most of these approaches are evaluated on small-scale and
low-resolution image datasets like MNIST, SVHN or CIFAR, which makes it
difficult to assess their applicability to the real world, and to compare them
among each other. We propose three open-set protocols that provide rich
datasets of natural images with different levels of similarity between known
and unknown classes. The protocols consist of subsets of ImageNet classes
selected to provide training and testing data closer to real-world scenarios.
Additionally, we propose a new validation metric that can be employed to assess
whether the training of deep learning models addresses both the classification
of known samples and the rejection of unknown samples. We use the protocols to
compare the performance of two baseline open-set algorithms to the standard
SoftMax baseline and find that the algorithms work well on negative samples
that have been seen during training, and partially on out-of-distribution
detection tasks, but drop performance in the presence of samples from
previously unseen unknown classes.
