  Visual Grounding (VG) methods in Visual Question Answering (VQA) attempt to
improve VQA performance by strengthening a model's reliance on
question-relevant visual information. The presence of such relevant information
in the visual input is typically assumed in training and testing. This
assumption, however, is inherently flawed when dealing with imperfect image
representations common in large-scale VQA, where the information carried by
visual features frequently deviates from expected ground-truth contents. As a
result, training and testing of VG-methods is performed with largely inaccurate
data, which obstructs proper assessment of their potential benefits. In this
study, we demonstrate that current evaluation schemes for VG-methods are
problematic due to the flawed assumption of availability of relevant visual
information. Our experiments show that these methods can be much more effective
when evaluation conditions are corrected. Code is provided on GitHub.
