  Vision and Language (VL) models have demonstrated remarkable zero-shot
performance in a variety of tasks. However, some aspects of complex language
understanding still remain a challenge. We introduce the collective notion of
Structured Vision&Language Concepts (SVLC) which includes object attributes,
relations, and states which are present in the text and visible in the image.
Recent studies have shown that even the best VL models struggle with SVLC. A
possible way of fixing this issue is by collecting dedicated datasets for
teaching each SVLC type, yet this might be expensive and time-consuming.
Instead, we propose a more elegant data-driven approach for enhancing VL
models' understanding of SVLCs that makes more effective use of existing VL
pre-training datasets and does not require any additional data. While automatic
understanding of image structure still remains largely unsolved, language
structure is much better modeled and understood, allowing for its effective
utilization in teaching VL models. In this paper, we propose various techniques
based on language structure understanding that can be used to manipulate the
textual part of off-the-shelf paired VL datasets. VL models trained with the
updated data exhibit a significant improvement of up to 15% in their SVLC
understanding with only a mild degradation in their zero-shot capabilities both
when training from scratch or fine-tuning a pre-trained model.
