  The challenge of fine-grained visual recognition often lies in discovering
the key discriminative regions. While such regions can be automatically
identified from a large-scale labeled dataset, a similar method might become
less effective when only a few annotations are available. In low data regimes,
a network often struggles to choose the correct regions for recognition and
tends to overfit spurious correlated patterns from the training data. To tackle
this issue, this paper proposes the self-boosting attention mechanism, a novel
method for regularizing the network to focus on the key regions shared across
samples and classes. Specifically, the proposed method first generates an
attention map for each training image, highlighting the discriminative part for
identifying the ground-truth object category. Then the generated attention maps
are used as pseudo-annotations. The network is enforced to fit them as an
auxiliary task. We call this approach the self-boosting attention mechanism
(SAM). We also develop a variant by using SAM to create multiple attention maps
to pool convolutional maps in a style of bilinear pooling, dubbed SAM-Bilinear.
Through extensive experimental studies, we show that both methods can
significantly improve fine-grained visual recognition performance on low data
regimes and can be incorporated into existing network architectures. The source
code is publicly available at: https://github.com/GANPerf/SAM
