  Deep learning models have recently demonstrated remarkable results in a
variety of tasks, which is why they are being increasingly applied in
high-stake domains, such as industry, medicine, and finance. Considering that
automatic predictions in these domains might have a substantial impact on the
well-being of a person, as well as considerable financial and legal
consequences to an individual or a company, all actions and decisions that
result from applying these models have to be accountable. Given that a
substantial amount of data that is collected in high-stake domains are in the
form of time series, in this paper we examine the current state of eXplainable
AI (XAI) methods with a focus on approaches for opening up deep learning black
boxes for the task of time series classification. Finally, our contribution
also aims at deriving promising directions for future work, to advance XAI for
deep learning on time series data.
