  Deep Learning based Automatic Speech Recognition (ASR) models are very
successful, but hard to interpret. To gain better understanding of how
Artificial Neural Networks (ANNs) accomplish their tasks, introspection methods
have been proposed. Adapting such techniques from computer vision to speech
recognition is not straight-forward, because speech data is more complex and
less interpretable than image data. In this work, we introduce
Gradient-adjusted Neuron Activation Profiles (GradNAPs) as means to interpret
features and representations in Deep Neural Networks. GradNAPs are
characteristic responses of ANNs to particular groups of inputs, which
incorporate the relevance of neurons for prediction. We show how to utilize
GradNAPs to gain insight about how data is processed in ANNs. This includes
different ways of visualizing features and clustering of GradNAPs to compare
embeddings of different groups of inputs in any layer of a given network. We
demonstrate our proposed techniques using a fully-convolutional ASR model.
