  Open-set object detection (OSOD) has recently gained attention. It is to
detect unknown objects while correctly detecting known objects. In this paper,
we first point out that the recent studies' formalization of OSOD, which
generalizes open-set recognition (OSR) and thus considers an unlimited variety
of unknown objects, has a fundamental issue. This issue emerges from the
difference between image classification and object detection, making it hard to
evaluate OSOD methods' performance properly. We then introduce a novel scenario
of OSOD, which considers known and unknown classes within a specified
super-class of object classes. This new scenario has practical applications and
is free from the above issue, enabling proper evaluation of OSOD performance
and probably making the problem more manageable. Finally, we experimentally
evaluate existing OSOD methods with the new scenario using multiple datasets,
showing that the current state-of-the-art OSOD methods attain limited
performance similar to a simple baseline method. The paper also presents a
taxonomy of OSOD that clarifies different problem formalizations. We hope our
study helps the community reconsider OSOD problems and progress in the right
direction.
