  Vision and language models (VLMs) have demonstrated remarkable zero-shot (ZS)
performance in a variety of tasks. However, recent works have shown that even
the best VLMs struggle to capture aspects of compositional scene understanding,
such as object attributes, relations, and action states. In contrast, obtaining
structured annotations, such as scene graphs (SGs), that could improve these
models is time-consuming and costly, and thus cannot be used on a large scale.
Here we ask whether small SG datasets can provide sufficient information for
enhancing structured understanding of pretrained VLMs. We show that it is
indeed possible to improve VLMs when learning from SGs by integrating
components that incorporate structured information into both visual and textual
representations. For the visual side, we incorporate a special "SG Component"
in the image transformer trained to predict SG information, while for the
textual side, we utilize SGs to generate fine-grained captions that highlight
different compositional aspects of the scene. Our method improves the
performance of several popular VLMs on multiple VL datasets with only a mild
degradation in ZS capabilities.
