  Deep learning has become a one-size-fits-all solution for technical and
business domains thanks to its flexibility and adaptability. It is implemented
using opaque models, which unfortunately undermines the outcome
trustworthiness. In order to have a better understanding of the behavior of a
system, particularly one driven by time series, a look inside a deep learning
model so-called posthoc eXplainable Artificial Intelligence (XAI) approaches,
is important. There are two major types of XAI for time series data, namely
model-agnostic and model-specific. Model-specific approach is considered in
this work. While other approaches employ either Class Activation Mapping (CAM)
or Attention Mechanism, we merge the two strategies into a single system,
simply called the Temporally Weighted Spatiotemporal Explainable Neural Network
for Multivariate Time Series (TSEM). TSEM combines the capabilities of RNN and
CNN models in such a way that RNN hidden units are employed as attention
weights for the CNN feature maps temporal axis. The result shows that TSEM
outperforms XCM. It is similar to STAM in terms of accuracy, while also
satisfying a number of interpretability criteria, including causality,
fidelity, and spatiotemporality.
