  Understanding the features learned by deep models is important from a model
trust perspective, especially as deep systems are deployed in the real world.
Most recent approaches for deep feature understanding or model explanation
focus on highlighting input data features that are relevant for classification
decisions. In this work, we instead take the perspective of relating deep
features to well-studied, hand-crafted features that are meaningful for the
application of interest. We propose a methodology and set of systematic
experiments for exploring deep features in this setting, where input feature
importance approaches for deep feature understanding do not apply. Our
experiments focus on understanding which hand-crafted and deep features are
useful for the classification task of interest, how robust these features are
for related tasks and how similar the deep features are to the meaningful
hand-crafted features. Our proposed method is general to many application areas
and we demonstrate its utility on orchestral music audio data.
