  While recent deep neural network models have achieved promising results on
the image captioning task, they rely largely on the availability of corpora
with paired image and sentence captions to describe objects in context. In this
work, we propose the Deep Compositional Captioner (DCC) to address the task of
generating descriptions of novel objects which are not present in paired
image-sentence datasets. Our method achieves this by leveraging large object
recognition datasets and external text corpora and by transferring knowledge
between semantically similar concepts. Current deep caption models can only
describe objects contained in paired image-sentence corpora, despite the fact
that they are pre-trained with large object recognition datasets, namely
ImageNet. In contrast, our model can compose sentences that describe novel
objects and their interactions with other objects. We demonstrate our model's
ability to describe novel concepts by empirically evaluating its performance on
MSCOCO and show qualitative results on ImageNet images of objects for which no
paired image-caption data exist. Further, we extend our approach to generate
descriptions of objects in video clips. Our results show that DCC has distinct
advantages over existing image and video captioning approaches for generating
descriptions of new objects in context.
