  Grounding language to visual relations is critical to various
language-and-vision applications. In this work, we tackle two fundamental
language-and-vision tasks: image-text matching and image captioning, and
demonstrate that neural scene graph generators can learn effective visual
relation features to facilitate grounding language to visual relations and
subsequently improve the two end applications. By combining relation features
with the state-of-the-art models, our experiments show significant improvement
on the standard Flickr30K and MSCOCO benchmarks. Our experimental results and
analysis show that relation features improve downstream models' capability of
capturing visual relations in end vision-and-language applications. We also
demonstrate the importance of learning scene graph generators with visually
relevant relations to the effectiveness of relation features.
