  Explanation and high-order reasoning capabilities are crucial for real-world
visual question answering with diverse levels of inference complexity (e.g.,
what is the dog that is near the girl playing with?) and important for users to
understand and diagnose the trustworthiness of the system. Current VQA
benchmarks on natural images with only an accuracy metric end up pushing the
models to exploit the dataset biases and cannot provide any interpretable
justification, which severally hinders advances in high-level question
answering. In this work, we propose a new HVQR benchmark for evaluating
explainable and high-order visual question reasoning ability with three
distinguishable merits: 1) the questions often contain one or two relationship
triplets, which requires the model to have the ability of multistep reasoning
to predict plausible answers; 2) we provide an explicit evaluation on a
multistep reasoning process that is constructed with image scene graphs and
commonsense knowledge bases; and 3) each relationship triplet in a large-scale
knowledge base only appears once among all questions, which poses challenges
for existing networks that often attempt to overfit the knowledge base that
already appears in the training set and enforces the models to handle unseen
questions and knowledge fact usage. We also propose a new knowledge-routed
modular network (KM-net) that incorporates the multistep reasoning process over
a large knowledge base into visual question reasoning. An extensive dataset
analysis and comparisons with existing models on the HVQR benchmark show that
our benchmark provides explainable evaluations, comprehensive reasoning
requirements and realistic challenges of VQA systems, as well as our KM-net's
superiority in terms of accuracy and explanation ability.
