  Visual Question Answering (VQA) is concerned with answering free-form
questions about an image. Since it requires a deep semantic and linguistic
understanding of the question and the ability to associate it with various
objects that are present in the image, it is an ambitious task and requires
multi-modal reasoning from both computer vision and natural language
processing. We propose Graphhopper, a novel method that approaches the task by
integrating knowledge graph reasoning, computer vision, and natural language
processing techniques. Concretely, our method is based on performing
context-driven, sequential reasoning based on the scene entities and their
semantic and spatial relationships. As a first step, we derive a scene graph
that describes the objects in the image, as well as their attributes and their
mutual relationships. Subsequently, a reinforcement learning agent is trained
to autonomously navigate in a multi-hop manner over the extracted scene graph
to generate reasoning paths, which are the basis for deriving answers. We
conduct an experimental study on the challenging dataset GQA, based on both
manually curated and automatically generated scene graphs. Our results show
that we keep up with a human performance on manually curated scene graphs.
Moreover, we find that Graphhopper outperforms another state-of-the-art scene
graph reasoning model on both manually curated and automatically generated
scene graphs by a significant margin.
