  In this paper, we propose an approach to improve image captioning solution
for images with novel objects that do not have caption labels in the training
dataset. We refer to our approach as Partially-Supervised Novel Object
Captioning (PS-NOC). PS-NOC is agnostic to model architecture, and primarily
focuses on the training approach that uses existing fully paired image-caption
data and the images with only the novel object detection labels (partially
paired data). We create synthetic paired captioning data for novel objects by
leveraging context from existing image-caption pairs. We then create
pseudo-label captions for partially paired images with novel objects, and use
this additional data to fine-tune the captioning model. We also propose a
variant of SCST within PS-NOC, called SCST-F1, that directly optimizes the
F1-score of novel objects. Using a popular captioning model (Up-Down) as
baseline, PS-NOC sets new state-of-the-art results on held-out MS COCO
out-of-domain test split, i.e., 85.9 F1-score and 103.8 CIDEr. This is an
improvement of 85.9 and 34.1 points respectively compared to baseline model
that does not use partially paired data during training. We also perform
detailed ablation studies to demonstrate the effectiveness of our approach.
