  Despite the remarkable accuracy of deep neural networks in object detection,
they are costly to train and scale due to supervision requirements.
Particularly, learning more object categories typically requires proportionally
more bounding box annotations. Weakly supervised and zero-shot learning
techniques have been explored to scale object detectors to more categories with
less supervision, but they have not been as successful and widely adopted as
supervised models. In this paper, we put forth a novel formulation of the
object detection problem, namely open-vocabulary object detection, which is
more general, more practical, and more effective than weakly supervised and
zero-shot approaches. We propose a new method to train object detectors using
bounding box annotations for a limited set of object categories, as well as
image-caption pairs that cover a larger variety of objects at a significantly
lower cost. We show that the proposed method can detect and localize objects
for which no bounding box annotation is provided during training, at a
significantly higher accuracy than zero-shot approaches. Meanwhile, objects
with bounding box annotation can be detected almost as accurately as supervised
methods, which is significantly better than weakly supervised baselines.
Accordingly, we establish a new state of the art for scalable object detection.
