  We propose a margin-based loss for tuning joint vision-language models so
that their gradient-based explanations are consistent with region-level
annotations provided by humans for relatively smaller grounding datasets. We
refer to this objective as Attention Mask Consistency (AMC) and demonstrate
that it produces superior visual grounding results than previous methods that
rely on using vision-language models to score the outputs of object detectors.
Particularly, a model trained with AMC on top of standard vision-language
modeling objectives obtains a state-of-the-art accuracy of 86.49% in the
Flickr30k visual grounding benchmark, an absolute improvement of 5.38% when
compared to the best previous model trained under the same level of
supervision. Our approach also performs exceedingly well on established
benchmarks for referring expression comprehension where it obtains 80.34%
accuracy in the easy test of RefCOCO+, and 64.55% in the difficult split. AMC
is effective, easy to implement, and is general as it can be adopted by any
vision-language model, and can use any type of region annotations.
