  Many top-performing image captioning models rely solely on object features
computed with an object detection model to generate image descriptions.
However, recent studies propose to directly use scene graphs to introduce
information about object relations into captioning, hoping to better describe
interactions between objects. In this work, we thoroughly investigate the use
of scene graphs in image captioning. We empirically study whether using
additional scene graph encoders can lead to better image descriptions and
propose a conditional graph attention network (C-GAT), where the image
captioning decoder state is used to condition the graph updates. Finally, we
determine to what extent noise in the predicted scene graphs influence caption
quality. Overall, we find no significant difference between models that use
scene graph features and models that only use object detection features across
different captioning metrics, which suggests that existing scene graph
generation models are still too noisy to be useful in image captioning.
Moreover, although the quality of predicted scene graphs is very low in
general, when using high quality scene graphs we obtain gains of up to 3.3
CIDEr compared to a strong Bottom-Up Top-Down baseline. We open source code to
reproduce all our experiments in
https://github.com/iacercalixto/butd-image-captioning.
