  Methods for teaching machines to answer visual questions have made
significant progress in recent years, but current methods still lack important
human capabilities, including integrating new visual classes and concepts in a
modular manner, providing explanations for the answers and handling new domains
without explicit examples. We propose a novel method that consists of two main
parts: generating a question graph representation, and an answering procedure,
guided by the abstract structure of the question graph to invoke an extendable
set of visual estimators. Training is performed for the language part and the
visual part on their own, but unlike existing schemes, the method does not
require any training using images with associated questions and answers. This
approach is able to handle novel domains (extended question types and new
object classes, properties and relations) as long as corresponding visual
estimators are available. In addition, it can provide explanations to its
answers and suggest alternatives when questions are not grounded in the image.
We demonstrate that this approach achieves both high performance and domain
extensibility without any questions-answers training.
