  Explainable AI (XAI) techniques have been widely used to help explain and
understand the output of deep learning models in fields such as image
classification and Natural Language Processing. Interest in using XAI
techniques to explain deep learning-based automatic speech recognition (ASR) is
emerging. but there is not enough evidence on whether these explanations can be
trusted. To address this, we adapt a state-of-the-art XAI technique from the
image classification domain, Local Interpretable Model-Agnostic Explanations
(LIME), to a model trained for a TIMIT-based phoneme recognition task. This
simple task provides a controlled setting for evaluation while also providing
expert annotated ground truth to assess the quality of explanations. We find a
variant of LIME based on time partitioned audio segments, that we propose in
this paper, produces the most reliable explanations, containing the ground
truth 96% of the time in its top three audio segments.
