  In this paper, we propose a novel approach that learns to sequentially attend
to different Convolutional Neural Networks (CNN) layers (i.e., ``what'' feature
abstraction to attend to) and different spatial locations of the selected
feature map (i.e., ``where'') to perform the task at hand. Specifically, at
each Recurrent Neural Network (RNN) step, both a CNN layer and localized
spatial region within it are selected for further processing. We demonstrate
the effectiveness of this approach on two computer vision tasks: (i)
image-based six degree of freedom camera pose regression and (ii) indoor scene
classification. Empirically, we show that combining the ``what'' and ``where''
aspects of attention improves network performance on both tasks. We evaluate
our method on standard benchmarks for camera localization (Cambridge, 7-Scenes,
and TUM-LSI) and for scene classification (MIT-67 Indoor Scenes). For camera
localization our approach reduces the median error by 18.8\% for position and
8.2\% for orientation (averaged over all scenes), and for scene classification
it improves the mean accuracy by 3.4\% over previous methods.
