  Can we leverage high-resolution information without the unsustainable
quadratic complexity to input scale? We propose Traversal Network (TNet), a
novel multi-scale hard-attention architecture, which traverses image
scale-space in a top-down fashion, visiting only the most informative image
regions along the way. TNet offers an adjustable trade-off between accuracy and
complexity, by changing the number of attended image locations. We compare our
model against hard-attention baselines on ImageNet, achieving higher accuracy
with less resources (FLOPs, processing time and memory). We further test our
model on fMoW dataset, where we process satellite images of size up to $896
\times 896$ px, getting up to $2.5$x faster processing compared to baselines
operating on the same resolution, while achieving higher accuracy as well. TNet
is modular, meaning that most classification models could be adopted as its
backbone for feature extraction, making the reported performance gains
orthogonal to benefits offered by existing optimized deep models. Finally,
hard-attention guarantees a degree of interpretability to our model's
predictions, without any extra cost beyond inference. Code is available at
$\href{https://github.com/Tpap/TNet}{github.com/Tpap/TNet}$.
