  Accurately answering a question about a given image requires combining
observations with general knowledge. While this is effortless for humans,
reasoning with general knowledge remains an algorithmic challenge. To advance
research in this direction a novel `fact-based' visual question answering
(FVQA) task has been introduced recently along with a large set of curated
facts which link two entities, i.e., two possible answers, via a relation.
Given a question-image pair, deep network techniques have been employed to
successively reduce the large set of facts until one of the two entities of the
final remaining fact is predicted as the answer. We observe that a successive
process which considers one fact at a time to form a local decision is
sub-optimal. Instead, we develop an entity graph and use a graph convolutional
network to `reason' about the correct answer by jointly considering all
entities. We show on the challenging FVQA dataset that this leads to an
improvement in accuracy of around 7% compared to the state of the art.
