  We aim to dismantle the prevalent black-box neural architectures used in
complex visual reasoning tasks, into the proposed eXplainable and eXplicit
Neural Modules (XNMs), which advance beyond existing neural module networks
towards using scene graphs --- objects as nodes and the pairwise relationships
as edges --- for explainable and explicit reasoning with structured knowledge.
XNMs allow us to pay more attention to teach machines how to "think",
regardless of what they "look". As we will show in the paper, by using scene
graphs as an inductive bias, 1) we can design XNMs in a concise and flexible
fashion, i.e., XNMs merely consist of 4 meta-types, which significantly reduce
the number of parameters by 10 to 100 times, and 2) we can explicitly trace the
reasoning-flow in terms of graph attentions. XNMs are so generic that they
support a wide range of scene graph implementations with various qualities. For
example, when the graphs are detected perfectly, XNMs achieve 100% accuracy on
both CLEVR and CLEVR CoGenT, establishing an empirical performance upper-bound
for visual reasoning; when the graphs are noisily detected from real-world
images, XNMs are still robust to achieve a competitive 67.5% accuracy on
VQAv2.0, surpassing the popular bag-of-objects attention models without graph
structures.
