  Deep neural networks have achieved great successes on the image captioning
task. However, most of the existing models depend heavily on paired
image-sentence datasets, which are very expensive to acquire. In this paper, we
make the first attempt to train an image captioning model in an unsupervised
manner. Instead of relying on manually labeled image-sentence pairs, our
proposed model merely requires an image set, a sentence corpus, and an existing
visual concept detector. The sentence corpus is used to teach the captioning
model how to generate plausible sentences. Meanwhile, the knowledge in the
visual concept detector is distilled into the captioning model to guide the
model to recognize the visual concepts in an image. In order to further
encourage the generated captions to be semantically consistent with the image,
the image and caption are projected into a common latent space so that they can
reconstruct each other. Given that the existing sentence corpora are mainly
designed for linguistic research and are thus with little reference to image
contents, we crawl a large-scale image description corpus of two million
natural sentences to facilitate the unsupervised image captioning scenario.
Experimental results show that our proposed model is able to produce quite
promising results without any caption annotations.
