  Object categories inherently form a hierarchy with different levels of
concept abstraction, especially for fine-grained categories. For example, birds
(Aves) can be categorized according to a four-level hierarchy of order, family,
genus, and species. This hierarchy encodes rich correlations among various
categories across different levels, which can effectively regularize the
semantic space and thus make prediction less ambiguous. However, previous
studies of fine-grained image recognition primarily focus on categories of one
certain level and usually overlook this correlation information. In this work,
we investigate simultaneously predicting categories of different levels in the
hierarchy and integrating this structured correlation information into the deep
neural network by developing a novel Hierarchical Semantic Embedding (HSE)
framework. Specifically, the HSE framework sequentially predicts the category
score vector of each level in the hierarchy, from highest to lowest. At each
level, it incorporates the predicted score vector of the higher level as prior
knowledge to learn finer-grained feature representation. During training, the
predicted score vector of the higher level is also employed to regularize label
prediction by using it as soft targets of corresponding sub-categories. To
evaluate the proposed framework, we organize the 200 bird species of the
Caltech-UCSD birds dataset with the four-level category hierarchy and construct
a large-scale butterfly dataset that also covers four level categories.
Extensive experiments on these two and the newly-released VegFru datasets
demonstrate the superiority of our HSE framework over the baseline methods and
existing competitors.
