  Fine-grained visual recognition typically depends on modeling subtle
difference from object parts. However, these parts often exhibit dramatic
visual variations such as occlusions, viewpoints, and spatial transformations,
making it hard to detect. In this paper, we present a novel attention-based
model to automatically, selectively and accurately focus on critical object
regions with higher importance against appearance variations. Given an image,
two different Convolutional Neural Networks (CNNs) are constructed, where the
outputs of two CNNs are correlated through bilinear pooling to simultaneously
focus on discriminative regions and extract relevant features. To capture
spatial distributions among the local regions with visual attention, soft
attention based spatial Long-Short Term Memory units (LSTMs) are incorporated
to realize spatially recurrent yet visually selective over local input
patterns. All the above intuitions equip our network with the following novel
model: two-stream CNN layers, bilinear pooling layer, spatial recurrent layer
with location attention are jointly trained via an end-to-end fashion to serve
as the part detector and feature extractor, whereby relevant features are
localized and extracted attentively. We show the significance of our network
against two well-known visual recognition tasks: fine-grained image
classification and person re-identification.
