  Hierarchical semantic classification requires the prediction of a taxonomy
tree instead of a single flat level of the tree, where both accuracies at
individual levels and consistency across levels matter. We can train
classifiers for individual levels, which has accuracy but not consistency, or
we can train only the finest level classification and infer higher levels,
which has consistency but not accuracy. Our key insight is that hierarchical
recognition should not be treated as multi-task classification, as each level
is essentially a different task and they would have to compromise with each
other, but be grounded on image segmentations that are consistent across
semantic granularities. Consistency can in fact improve accuracy. We build upon
recent work on learning hierarchical segmentation for flat-level recognition,
and extend it to hierarchical recognition. It naturally captures the intuition
that fine-grained recognition requires fine image segmentation whereas
coarse-grained recognition requires coarse segmentation; they can all be
integrated into one recognition model that drives fine-to-coarse internal
visual parsing.Additionally, we introduce a Tree-path KL Divergence loss to
enforce consistent accurate predictions across levels. Our extensive
experimentation and analysis demonstrate our significant gains on predicting an
accurate and consistent taxonomy tree.
