  We investigate the problem of reducing mistake severity for fine-grained
classification. Fine-grained classification can be challenging, mainly due to
the requirement of domain expertise for accurate annotation. However, humans
are particularly adept at performing coarse classification as it requires
relatively low levels of expertise. To this end, we present a novel approach
for Post-Hoc Correction called Hierarchical Ensembles (HiE) that utilizes label
hierarchy to improve the performance of fine-grained classification at
test-time using the coarse-grained predictions. By only requiring the parents
of leaf nodes, our method significantly reduces avg. mistake severity while
improving top-1 accuracy on the iNaturalist-19 and tieredImageNet-H datasets,
achieving a new state-of-the-art on both benchmarks. We also investigate the
efficacy of our approach in the semi-supervised setting. Our approach brings
notable gains in top-1 accuracy while significantly decreasing the severity of
mistakes as training data decreases for the fine-grained classes. The
simplicity and post-hoc nature of HiE renders it practical to be used with any
off-the-shelf trained model to improve its predictions further.
