  Vision and Language Pretraining has become the prevalent approach for
tackling multimodal downstream tasks. The current trend is to move towards ever
larger models and pretraining datasets. This computational headlong rush does
not seem reasonable in the long term to move toward sustainable solutions, and
de facto excludes academic laboratories with limited resources. In this work,
we propose a new framework, dubbed ViCHA, that efficiently exploits the input
data to boost the learning by: (a) a new hierarchical cross-modal alignment
loss, (b) new self-supervised scheme based on masked image modeling, (c)
leveraging image-level annotations, called Visual Concepts, obtained with
existing foundation models such as CLIP to boost the performance of the image
encoder. Although pretrained on four times less data, our ViCHA strategy
outperforms other approaches on several downstream tasks such as Image-Text
Retrieval, VQA, Visual Reasoning, Visual Entailment and Visual Grounding. The
code will be made publicly available here: https://github.com/mshukor/ViCHA
