  Recent advances in eXplainable AI (XAI) have provided new insights into how
models for vision, language, and tabular data operate. However, few approaches
exist for understanding speech models. Existing work focuses on a few spoken
language understanding (SLU) tasks, and explanations are difficult to interpret
for most users. We introduce a new approach to explain speech classification
models. We generate easy-to-interpret explanations via input perturbation on
two information levels. 1) Word-level explanations reveal how each word-related
audio segment impacts the outcome. 2) Paralinguistic features (e.g., prosody
and background noise) answer the counterfactual: ``What would the model
prediction be if we edited the audio signal in this way?'' We validate our
approach by explaining two state-of-the-art SLU models on two speech
classification tasks in English and Italian. Our findings demonstrate that the
explanations are faithful to the model's inner workings and plausible to
humans. Our method and findings pave the way for future research on
interpreting speech models.
