  Convolutional Neural Networks (CNNs) currently achieve state-of-the-art
accuracy in image classification. With a growing number of classes, the
accuracy usually drops as the possibilities of confusion increase.
Interestingly, the class confusion patterns follow a hierarchical structure
over the classes. We present visual-analytics methods to reveal and analyze
this hierarchy of similar classes in relation with CNN-internal data. We found
that this hierarchy not only dictates the confusion patterns between the
classes, it furthermore dictates the learning behavior of CNNs. In particular,
the early layers in these networks develop feature detectors that can separate
high-level groups of classes quite well, even after a few training epochs. In
contrast, the latter layers require substantially more epochs to develop
specialized feature detectors that can separate individual classes. We
demonstrate how these insights are key to significant improvement in accuracy
by designing hierarchy-aware CNNs that accelerate model convergence and
alleviate overfitting. We further demonstrate how our methods help in
identifying various quality issues in the training data.
