  Recently, image captioning has aroused great interest in both academic and
industrial worlds. Most existing systems are built upon large-scale datasets
consisting of image-sentence pairs, which, however, are time-consuming to
construct. In addition, even for the most advanced image captioning systems, it
is still difficult to realize deep image understanding. In this work, we
achieve unpaired image captioning by bridging the vision and the language
domains with high-level semantic information. The motivation stems from the
fact that the semantic concepts with the same modality can be extracted from
both images and descriptions. To further improve the quality of captions
generated by the model, we propose the Semantic Relationship Explorer, which
explores the relationships between semantic concepts for better understanding
of the image. Extensive experiments on MSCOCO dataset show that we can generate
desirable captions without paired datasets. Furthermore, the proposed approach
boosts five strong baselines under the paired setting, where the most
significant improvement in CIDEr score reaches 8%, demonstrating that it is
effective and generalizes well to a wide range of models.
