  Deep neural networks for image classification typically consists of a
convolutional feature extractor followed by a fully connected classifier
network. The predicted and the ground truth labels are represented as one hot
vectors. Such a representation assumes that all classes are equally dissimilar.
However, classes have visual similarities and often form a hierarchy. Learning
this latent hierarchy explicitly in the architecture could provide invaluable
insights. We propose an alternate architecture to the classifier network called
the Latent Hierarchy (LH) Classifier and an end to end learned Class2Str
mapping which discovers a latent hierarchy of the classes. We show that for
some of the best performing architectures on CIFAR and Imagenet datasets, the
proposed replacement and training by LH classifier recovers the accuracy, with
a fraction of the number of parameters in the classifier part. Compared to the
previous work of HDCNN, which also learns a 2 level hierarchy, we are able to
learn a hierarchy at an arbitrary number of levels as well as obtain an
accuracy improvement on the Imagenet classification task over them. We also
verify that many visually similar classes are grouped together, under the
learnt hierarchy.
