  Open-set Recognition (OSR) aims to identify test samples whose classes are
not seen during the training process. Recently, Unified Open-set Recognition
(UOSR) has been proposed to reject not only unknown samples but also known but
wrongly classified samples, which tends to be more practical in real-world
applications. The UOSR draws little attention since it is proposed, but we find
sometimes it is even more practical than OSR in the real world applications, as
evaluation results of known but wrongly classified samples are also wrong like
unknown samples. In this paper, we deeply analyze the UOSR task under different
training and evaluation settings to shed light on this promising research
direction. For this purpose, we first evaluate the UOSR performance of several
OSR methods and show a significant finding that the UOSR performance
consistently surpasses the OSR performance by a large margin for the same
method. We show that the reason lies in the known but wrongly classified
samples, as their uncertainty distribution is extremely close to unknown
samples rather than known and correctly classified samples. Second, we analyze
how the two training settings of OSR (i.e., pre-training and outlier exposure)
influence the UOSR. We find although they are both beneficial for
distinguishing known and correctly classified samples from unknown samples,
pre-training is also helpful for identifying known but wrongly classified
samples while outlier exposure is not. In addition to different training
settings, we also formulate a new evaluation setting for UOSR which is called
few-shot UOSR, where only one or five samples per unknown class are available
during evaluation to help identify unknown samples. We propose FS-KNNS for the
few-shot UOSR to achieve state-of-the-art performance under all settings.
