  Small inter-class and large intra-class variations are the main challenges in
fine-grained visual classification. Objects from different classes share
visually similar structures and objects in the same class can have different
poses and viewpoints. Therefore, the proper extraction of discriminative local
features (e.g. bird's beak or car's headlight) is crucial. Most of the recent
successes on this problem are based upon the attention models which can
localize and attend the local discriminative objects parts. In this work, we
propose a training method for visual attention networks, Coarse2Fine, which
creates a differentiable path from the input space to the attended feature
maps. Coarse2Fine learns an inverse mapping function from the attended feature
maps to the informative regions in the raw image, which will guide the
attention maps to better attend the fine-grained features. We show Coarse2Fine
and orthogonal initialization of the attention weights can surpass the
state-of-the-art accuracies on common fine-grained classification tasks.
