  Image classification has been one of the most popular tasks in Deep Learning,
seeing an abundance of impressive implementations each year. However, there is
a lot of criticism tied to promoting complex architectures that continuously
push performance metrics higher and higher. Robustness tests can uncover
several vulnerabilities and biases which go unnoticed during the typical model
evaluation stage. So far, model robustness under distribution shifts has mainly
been examined within carefully curated datasets. Nevertheless, such approaches
do not test the real response of classifiers in the wild, e.g. when uncurated
web-crawled image data of corresponding classes are provided. In our work, we
perform fine-grained classification on closely related categories, which are
identified with the help of hierarchical knowledge. Extensive experimentation
on a variety of convolutional and transformer-based architectures reveals model
robustness in this novel setting. Finally, hierarchical knowledge is again
employed to evaluate and explain misclassifications, providing an
information-rich evaluation scheme adaptable to any classifier.
