  Without the demand of training in reality, humans can easily detect a known
concept simply based on its language description. Empowering deep learning with
this ability undoubtedly enables the neural network to handle complex vision
tasks, e.g., object detection, without collecting and annotating real images.
To this end, this paper introduces a novel challenging learning paradigm
Imaginary-Supervised Object Detection (ISOD), where neither real images nor
manual annotations are allowed for training object detectors. To resolve this
challenge, we propose ImaginaryNet, a framework to synthesize images by
combining pretrained language model and text-to-image synthesis model. Given a
class label, the language model is used to generate a full description of a
scene with a target object, and the text-to-image model deployed to generate a
photo-realistic image. With the synthesized images and class labels, weakly
supervised object detection can then be leveraged to accomplish ISOD. By
gradually introducing real images and manual annotations, ImaginaryNet can
collaborate with other supervision settings to further boost detection
performance. Experiments show that ImaginaryNet can (i) obtain about 70%
performance in ISOD compared with the weakly supervised counterpart of the same
backbone trained on real data, (ii) significantly improve the baseline while
achieving state-of-the-art or comparable performance by incorporating
ImaginaryNet with other supervision settings.
