  Visual understanding requires comprehending complex visual relations between
objects within a scene. Here, we seek to characterize the computational demands
for abstract visual reasoning. We do this by systematically assessing the
ability of modern deep convolutional neural networks (CNNs) to learn to solve
the "Synthetic Visual Reasoning Test" (SVRT) challenge, a collection of
twenty-three visual reasoning problems. Our analysis reveals a novel taxonomy
of visual reasoning tasks, which can be primarily explained by both the type of
relations (same-different vs. spatial-relation judgments) and the number of
relations used to compose the underlying rules. Prior cognitive neuroscience
work suggests that attention plays a key role in humans' visual reasoning
ability. To test this hypothesis, we extended the CNNs with spatial and
feature-based attention mechanisms. In a second series of experiments, we
evaluated the ability of these attention networks to learn to solve the SVRT
challenge and found the resulting architectures to be much more efficient at
solving the hardest of these visual reasoning tasks. Most importantly, the
corresponding improvements on individual tasks partially explained our novel
taxonomy. Overall, this work provides an granular computational account of
visual reasoning and yields testable neuroscience predictions regarding the
differential need for feature-based vs. spatial attention depending on the type
of visual reasoning problem.
