  Visual Grounding (VG) in VQA refers to a model's proclivity to infer answers
based on question-relevant image regions. Conceptually, VG identifies as an
axiomatic requirement of the VQA task. In practice, however, DNN-based VQA
models are notorious for bypassing VG by way of shortcut (SC) learning without
suffering obvious performance losses in standard benchmarks. To uncover the
impact of SC learning, Out-of-Distribution (OOD) tests have been proposed that
expose a lack of VG with low accuracy. These tests have since been at the
center of VG research and served as basis for various investigations into VG's
impact on accuracy. However, the role of VG in VQA still remains not fully
understood and has not yet been properly formalized.
  In this work, we seek to clarify VG's role in VQA by formalizing it on a
conceptual level. We propose a novel theoretical framework called "Visually
Grounded Reasoning" (VGR) that uses the concepts of VG and Reasoning to
describe VQA inference in ideal OOD testing. By consolidating fundamental
insights into VG's role in VQA, VGR helps to reveal rampant VG-related SC
exploitation in OOD testing, which explains why the relationship between VG and
OOD accuracy has been difficult to define. Finally, we propose an approach to
create OOD tests that properly emphasize a requirement for VG, and show how to
improve performance on them.
