  Visual object recognition systems need to generalize from a set of 2D
training views to novel views. The question of how the human visual system can
generalize to novel views has been studied and modeled in psychology, computer
vision, and neuroscience. Modern deep learning architectures for object
recognition generalize well to novel views, but the mechanisms are not well
understood. In this paper, we characterize the ability of common deep learning
architectures to generalize to novel views. We formulate this as a supervised
classification task where labels correspond to unique 3D objects and examples
correspond to 2D views of the objects at different 3D orientations. We consider
three common models of generalization to novel views: (i) full 3D
generalization, (ii) pure 2D matching, and (iii) matching based on a linear
combination of views. We find that deep models generalize well to novel views,
but they do so in a way that differs from all these existing models.
Extrapolation to views beyond the range covered by views in the training set is
limited, and extrapolation to novel rotation axes is even more limited,
implying that the networks do not infer full 3D structure, nor use linear
interpolation. Yet, generalization is far superior to pure 2D matching. These
findings help with designing datasets with 2D views required to achieve 3D
generalization. Code to reproduce our experiments is publicly available:
https://github.com/shoaibahmed/investigating_3d_generalization.git
