  Explainable Artificial Intelligence (XAI) methods are typically deployed to
explain and debug black-box machine learning models. However, most proposed XAI
methods are black-boxes themselves and designed for images. Thus, they rely on
visual interpretability to evaluate and prove explanations. In this work, we
apply XAI methods previously used in the image and text-domain on time series.
We present a methodology to test and evaluate various XAI methods on time
series by introducing new verification techniques to incorporate the temporal
dimension. We further conduct preliminary experiments to assess the quality of
selected XAI method explanations with various verification methods on a range
of datasets and inspecting quality metrics on it. We demonstrate that in our
initial experiments, SHAP works robust for all models, but others like
DeepLIFT, LRP, and Saliency Maps work better with specific architectures.
