  Naturally, fine-grained recognition, e.g., vehicle identification or bird
classification, has specific hierarchical labels, where fine categories are
always harder to be classified than coarse categories. However, most of the
recent deep learning based methods neglect the semantic structure of
fine-grained objects and do not take advantage of the traditional fine-grained
recognition techniques (e.g. coarse-to-fine classification). In this paper, we
propose a novel framework with a two-branch network (coarse branch and fine
branch), i.e., semantic bilinear pooling, for fine-grained recognition with a
hierarchical label tree. This framework can adaptively learn the semantic
information from the hierarchical levels. Specifically, we design a generalized
cross-entropy loss for the training of the proposed framework to fully exploit
the semantic priors via considering the relevance between adjacent levels and
enlarge the distance between samples of different coarse classes. Furthermore,
our method leverages only the fine branch when testing so that it adds no
overhead to the testing time. Experimental results show that our proposed
method achieves state-of-the-art performance on four public datasets.
