  We revisit a particular visual grounding method: the "Image Retrieval Using
Scene Graphs" (IRSG) system of Johnson et al. (2015). Our experiments indicate
that the system does not effectively use its learned object-relationship
models. We also look closely at the IRSG dataset, as well as the widely used
Visual Relationship Dataset (VRD) that is adapted from it. We find that these
datasets exhibit biases that allow methods that ignore relationships to perform
relatively well. We also describe several other problems with the IRSG dataset,
and report on experiments using a subset of the dataset in which the biases and
other problems are removed. Our studies contribute to a more general effort:
that of better understanding what machine learning methods that combine
language and vision actually learn and what popular datasets actually test.
