  Explainable Artificial Intelligence (XAI) is targeted at understanding how
models perform feature selection and derive their classification decisions.
This paper explores post-hoc explanations for deep neural networks in the audio
domain. Notably, we present a novel Open Source audio dataset consisting of
30,000 audio samples of English spoken digits which we use for classification
tasks on spoken digits and speakers' biological sex. We use the popular XAI
technique Layer-wise Relevance Propagation (LRP) to identify relevant features
for two neural network architectures that process either waveform or
spectrogram representations of the data. Based on the relevance scores obtained
from LRP, hypotheses about the neural networks' feature selection are derived
and subsequently tested through systematic manipulations of the input data.
Further, we take a step beyond visual explanations and introduce audible
heatmaps. We demonstrate the superior interpretability of audible explanations
over visual ones in a human user study.
