  The mainstream image captioning models rely on Convolutional Neural Network
(CNN) image features to generate captions via recurrent models. Recently, image
scene graphs have been used to augment captioning models so as to leverage
their structural semantics, such as object entities, relationships and
attributes. Several studies have noted that the naive use of scene graphs from
a black-box scene graph generator harms image captioning performance and that
scene graph-based captioning models have to incur the overhead of explicit use
of image features to generate decent captions. Addressing these challenges, we
propose \textbf{SG2Caps}, a framework that utilizes only the scene graph labels
for competitive image captioning performance. The basic idea is to close the
semantic gap between the two scene graphs - one derived from the input image
and the other from its caption. In order to achieve this, we leverage the
spatial location of objects and the Human-Object-Interaction (HOI) labels as an
additional HOI graph. SG2Caps outperforms existing scene graph-only captioning
models by a large margin, indicating scene graphs as a promising representation
for image captioning. Direct utilization of scene graph labels avoids expensive
graph convolutions over high-dimensional CNN features resulting in 49% fewer
trainable parameters. Our code is available at:
https://github.com/Kien085/SG2Caps
