  Most of the currently existing vision and language pre-training (VLP) methods
have mainly focused on how to extract and align vision and text features. In
contrast to the mainstream VLP methods, we highlight that two routinely applied
steps during pre-training have crucial impact on the performance of the
pre-trained model: in-batch hard negative sampling for image-text matching
(ITM) and assigning the large masking probability for the masked language
modeling (MLM). After empirically showing the unexpected effectiveness of above
two steps, we systematically devise our GRIT-VLP, which adaptively samples
mini-batches for more effective mining of hard negative samples for ITM while
maintaining the computational cost for pre-training. Our method consists of
three components: 1) GRouped mIni-baTch sampling (GRIT) strategy that collects
similar examples in a mini-batch, 2) ITC consistency loss for improving the
mining ability, and 3) enlarged masking probability for MLM. Consequently, we
show our GRIT-VLP achieves a new state-of-the-art performance on various
downstream tasks with much less computational cost. Furthermore, we demonstrate
that our model is essentially in par with ALBEF, the previous state-of-the-art,
only with one-third of training epochs on the same training data. Code is
available at https://github.com/jaeseokbyun/GRIT-VLP.
