  Spatial reasoning is an important component of human intelligence. We can
imagine the shapes of 3D objects and reason about their spatial relations by
merely looking at their three-view line drawings in 2D, with different levels
of competence. Can deep networks be trained to perform spatial reasoning tasks?
How can we measure their "spatial intelligence"? To answer these questions, we
present the SPARE3D dataset. Based on cognitive science and psychometrics,
SPARE3D contains three types of 2D-3D reasoning tasks on view consistency,
camera pose, and shape generation, with increasing difficulty. We then design a
method to automatically generate a large number of challenging questions with
ground truth answers for each task. They are used to provide supervision for
training our baseline models using state-of-the-art architectures like ResNet.
Our experiments show that although convolutional networks have achieved
superhuman performance in many visual learning tasks, their spatial reasoning
performance on SPARE3D tasks is either lower than average human performance or
even close to random guesses. We hope SPARE3D can stimulate new problem
formulations and network designs for spatial reasoning to empower intelligent
robots to operate effectively in the 3D world via 2D sensors. The dataset and
code are available at https://ai4ce.github.io/SPARE3D.
